Author Topic: no output from sort-bed for large files  (Read 4089 times)

sjn

  • Administrator
  • Jr. Member
  • *****
  • Posts: 72
Re: no output from sort-bed for large files
« Reply #15 on: January 13, 2014, 09:54:00 AM »
OK, thanks.  We'll plan to improve this in v2.4.1.  See https://github.com/bedops/bedops/issues/34
« Last Edit: January 24, 2014, 05:07:04 AM by sjn »

sjn

  • Administrator
  • Jr. Member
  • *****
  • Posts: 72
Re: no output from sort-bed for large files
« Reply #16 on: February 10, 2014, 01:48:41 PM »
This is functional in our 2.4.1 development version.

git pull
git checkout v2p4p1

then, clean things and rebuild/install.
make very_clean && make && make install

« Last Edit: February 10, 2014, 02:34:01 PM by sjn »

pachkov

  • Newbie
  • *
  • Posts: 14
Re: no output from sort-bed for large files
« Reply #17 on: February 18, 2014, 01:46:43 AM »
Hi!

Thank you for this feature. Indeed it keeps temporary files in specified directory!
However I have got problem that at very end it crashes:

*** glibc detected *** ./sort-bed: double free or corruption (out): 0x000000002aa12470 ***
...

The final bed file is fine but in the temporary dir still contains some temp files left (in my case 3 out of 14).
Any suggestions?

Best,

Mikhail

sjn

  • Administrator
  • Jr. Member
  • *****
  • Posts: 72
Re: no output from sort-bed for large files
« Reply #18 on: February 18, 2014, 07:48:41 AM »
OK, we'll take a look, thanks.  How are you using things?  --tmpdir or with TMPDIR environmental variable?

Shane

pachkov

  • Newbie
  • *
  • Posts: 14
Re: no output from sort-bed for large files
« Reply #19 on: February 18, 2014, 07:53:09 AM »
I used --tmpdir. I will test latter $TMPDIR.

BTW, if I can help somehow with tracing down this bug, please tell me. I will be glad to help.

Best,

Mikhail

sjn

  • Administrator
  • Jr. Member
  • *****
  • Posts: 72
Re: no output from sort-bed for large files
« Reply #20 on: February 18, 2014, 09:09:52 AM »
Okay, I'm seeing a double-free now.  I'll look in detail.

sjn

  • Administrator
  • Jr. Member
  • *****
  • Posts: 72
Re: no output from sort-bed for large files
« Reply #21 on: February 18, 2014, 10:09:37 AM »
This has been fixed in a commit to version v2p4p1 now.
Feel free to:
git pull
make
make install

and give this a whirl again.  Sorry for the troubles and thanks for reporting the problem.

Shane
« Last Edit: February 18, 2014, 05:46:23 PM by sjn »

pachkov

  • Newbie
  • *
  • Posts: 14
Re: no output from sort-bed for large files
« Reply #22 on: February 19, 2014, 02:20:36 AM »
Hi Shane,

Tried v2p4p1. It finished without any error messages, sorted bed looks fine, tmpdir is empty in the end. Now started run without --tmpdir but with set $TMPDIR. It seems to work correctly.

Good work! Thanks!

Best,

Mikhail

sjn

  • Administrator
  • Jr. Member
  • *****
  • Posts: 72
Re: no output from sort-bed for large files
« Reply #23 on: February 22, 2014, 10:16:51 AM »
Great.
We're getting close to releasing v2p4p1.  I've made some new changes that should make things a bit more robust.  Previously, if you specified a --tmpdir that pointed at a file (not a directory), things would run and give crazy results.  I've cleaned up some other things too.  If you wouldn't mind testing again, that would be greatly appreciated!

There is one more piece that is a bit disturbing.  If something fails (such as running out of disk space in your case), sort-bed won't clean up any generated tmp files for you.  I'm still looking at how that might be accomplished.  It's a pain in C to deal with these sorts of things.  It can be done, of course, but I run the risk of introducing more bugs.  Thinking about how I might robustly test some things.  Anyway, not sure if this will happen or if I'll chalk it up to "well, we keep those files around so you can look over things upon failure"...

Shane

pachkov

  • Newbie
  • *
  • Posts: 14
Re: no output from sort-bed for large files
« Reply #24 on: February 25, 2014, 03:02:54 AM »
Hi Shane,

I have downloaded yesterday v2p4p1, compiled and tested sort-bed. It works flawless. I tested botth with --tmpdir and with $TMPDIR.

I agree that garbage left after program crash is a big headache. My only suggestion is to add warning message at the start in case --tmpdir or $TMPDIR is set. Something telling that non standard tmp dir will be used and potentially some files might not be cleaned after program finished.

Best,

Mikhail

sjn

  • Administrator
  • Jr. Member
  • *****
  • Posts: 72
Re: no output from sort-bed for large files
« Reply #25 on: February 25, 2014, 06:25:32 AM »
Awesome - thank you!

We have officially released version 2.4.1.
« Last Edit: February 26, 2014, 08:10:03 AM by sjn »