GMOS IFU data reduction scripts (updated Oct 2016)

Home Forums Gemini Data Reduction GMOS IFU data reduction scripts (updated Oct 2016)

This topic contains 24 replies, has 4 voices, and was last updated by  jturner 8 months ago.

Viewing 15 posts - 1 through 15 (of 25 total)
  • Author
    Posts
  • #482

    jturner
    Keymaster

    Here’s our own modified version of the GMOS IRAF package, with a handful of prototype improvements for reducing IFU data. We’ll unimaginatively refer to it as “IFUDR GMOS”, after the name someone assigned to our software repository. For those of you who have previously come across Bryan’s “ifuproc” scripts, this includes several of the same features, but not the top-level ifuproc itself or its QE correction, for which there’s now an official Gemini version under development (to be merged later).

    You can find some further description and installation details in the file README.

    We realize it would help to have a reduction example to go with this, but James is still working on the project for which he recently re-did these scripts and it seems better to post the code in the meantime than to wait.

    James will also upload his “pyfu” datacube mosaicking package separately (I’m just in the middle of adding a spatial registration method based on cross-correlation, which it would be good to include).

    Please post any questions about this unofficial version here, rather than through the helpdesk.

    Cheers,

    James & Bryan.

    Update: these attachments are old; see below for a newer version.

    0
    0
    • This topic was modified 3 years, 7 months ago by Emma :) Emma :).
    • This topic was modified 2 years, 10 months ago by Emma :) Emma :).
    • This topic was modified 2 years, 10 months ago by  jturner.
    • This topic was modified 2 years, 10 months ago by  jturner.
    • This topic was modified 9 months ago by achene achene.
    • This topic was modified 8 months, 3 weeks ago by  jturner.
    • This topic was modified 8 months, 3 weeks ago by  jturner.
    #629

    jturner
    Keymaster

    Update: I’ve added the mosaicking scripts here.

    0
    0
    • This reply was modified 3 years, 1 month ago by  jturner.
    #676

    jturner
    Keymaster

    Here’s a newer version of ifudrgmos, which corrects a bug in gmosaic for GMOS-N data, solves the replacement of incorrect regions by gemfix with auto-initialized DQ, mends the variance propagation in gscalibrate and adds the following features: approximate variance & units of flux/arcsec^2 in gfcube, an option to re-interpolate chip gaps after extraction and a stand-alone script for growing DQ>0 into neighbouring pixels.

    Cheers,

    James.

    0
    0
    • This reply was modified 2 years, 10 months ago by  jturner.
    #680

    jturner
    Keymaster

    It seems I forgot to mention before that this unofficial version does not have updated help pages, just a note of what changed in the README file. As and when these versions get merged into the official package, the help (and our tests etc.) will be updated accordingly.

    0
    0
    #693

    dara norman
    Participant

    Hi James,
    I seem to be having problems running ifudr right off the bat! I am not sure if the problem is in the installation of you fork OR in how I am using gfreduce itself, but I am trying to extract the apertures from the flat of a one slit mode ifu observation using gfreduce. I have Ureka installed, along side my regular IRAF which is also 2.16.
    I get through the bias and overscan subtraction but when I get to GFextract -> Gmosaic I get an error that:
    ERROR: Cannot open file (/tmp///.IMT501)
    “imdelete (tmptile, verify-, >& “dev$null”)”
    line 1189: gmos$gmosaic.cl
    called as: `gmosaic (inimages=tmpin9450x, outimages=tmppaste9450t, outpref=, fl_paste=no, fl_vardq=no, fl_fixpix=yes, fl_clean=no, geointer=linear, gap=default, bpmfile=gmos$data/chipgaps.dat, statsec=default, obsmode=IMAGE, sci_ext=SCI, var_ext=VAR, dq_ext=DQ, mdf_ext=MDF, key_detsec=DETSEC, key_datsec=DATASEC, key_ccdsum=CCDSUM, key_obsmode=OBSMODE, logfile=gmos.log, fl_real=no, verbose=yes)’
    “logfile=l_logfile, fl_real-, verbose=l_verbose)”
    line 642: gmos$gfextract.cl

    If I just look for the file /tmp///.IMT501 with an ls, it is there so I don’t understand what the problem is.
    I followed the instructions in your readme file for installation of the ifudr
    DARA

    0
    0
    #694

    jturner
    Keymaster

    Hi Dara,

    I believe .IMT501 is the Unix socket via which IRAF is trying to communicate with DS9 or XImtool. What’s unclear to me is why this should pop up in gmosaic, which doesn’t actually display anything…

    Mark spotted your post and mentioned that he may have seen this before due to a login.cl or uparm problem. We’re not sure off hand, but it might be that “/tmp///.IMT501” got picked up erroneously in place of the correct filename.

    Could you first try creating a new directory and running “mkiraf” there after ur_setup, please, then try it again? Or if you have lots of FITS files and don’t want to copy them around, just move your login.cl, loginuser.cl and uparm somewhere else and then run mkiraf.

    Also, are you running everything (including DS9) as the same user and working in a directory owned by you, with normal permissions?

    Thanks,

    James.

    0
    0
    #695

    dara norman
    Participant

    Hi James, I will try your suggestions ASAP. Hopefully before the end of the week. I am finishing up with my REU student this week. DARA

    0
    0
    #696

    dara norman
    Participant

    Hi James, OK I have things working now. The problem was with running ur_setup in the right order for things to work properly under Ureka. I think the instructions could be more clear… Thanks for the quick responses. DARA

    0
    0
    #697

    dara norman
    Participant

    Hi again, Well I didn’t get so far into my reduction… I am running into a problem with gfresponse. I am using a twilight flat to make a response curve and I run into the error:
    ERROR: ambiguous parameter a' within<searchpath>’
    “# Transform this “index image” as usual onto the same co-ordinates a …”
    line 265: gmos$gfresponse.cl
    called as: `gfresponse (mode=h)’

    When I have a look at gfresponse.cl, my limited knowledge of these things suggests that the culprit seems to be the line of code just before line 265 , where there is the loop:
    for (i=j1; i<=nslit; i+=1) {
    j=j+1
    imexpr(“I+0*a”, idxim//”[“//l_sci_ext//”,”//j//”,append]”,
    a=l_inimage//”[“//l_sci_ext//”,”//j//”]”)
    }
    However, I have no idea how to fix this error. I am surprised that one can even have an equation in this context.
    Thanks for any/all help you can give.
    DARA

    0
    0
    #698

    jturner
    Keymaster

    Hello,

    I’m glad your first problem went away. Which instructions do you think could be clearer? Any particular section?

    Are you using IRAF CL or PyRAF here? Note that I mention in the README that my modified gfresponse currently isn’t expected to work with twilight flats — I would suggest trying it with just the GCal flat unless you’re really concerned about overall spatial flatness at the level of, say, 1%. Do you still see this error without the twilight?

    Cheers,

    James.

    0
    0
    #699

    dara norman
    Participant

    Hi James,
    I am very sorry for the delay in responding and will try to do better.
    Apologies but I think I mis-stated my problem. First I am indeed in Iraf cl. I am using gfresponse on the gcal flat, but it is true that I was using the twilight flat to do an illumination correction, i.e.
    gfresponse gcalflat output sky=’twilight_flat’

    I tried just leaving off the ‘sky=twilight_flat’ part and I get the same error. I also tried using sky= some other gcalflat, but I get the same error as above. Therefore, I don’t think the error is in the fact that I am using a twilight for the correction. As I mentioned, I am surprised that one can have an equation in this context in the code, so is there something that needs to be set that I am not doing?

    Thanks for any and all help. DARA

    0
    0
    #700

    jturner
    Keymaster

    Hi Dara,

    Sorry, I see. I can reproduce that in CL, though for some reason IRAF tells me a different line number (it usually gets them wrong anyway). I have to admit I’ve only really been testing my code in PyRAF, until such time as it gets merged into the official package.

    This is pretty silly CL behaviour, but you can “fix” it simply by removing the “a=” on L262 so that the last argument to imexpr just becomes positional. Or you can use PyRAF… Anyway, thanks for the report; I’ll change it.

    The odd equation is just a means of creating an intermediate image with the same size & properties as the input (that’s what the *0 is for) but with column numbers in it (that’s the I). I had to do some trickery to get things transformed backwards from linear wavelength to the non-linear extracted co-ordinates. I may actually have a better way of doing this in future, after some work on QE correction in the official package, though I think what’s here works well enough (apart from this crash).

    Cheers,

    James.

    0
    0
    #701

    jturner
    Keymaster

    I may be able to get my latest gfcube version into an upcoming release of the official Gemini package and I’d just like to collect any user feedback on the new variance propagation & flux units behaviour.

    Because gfcube resamples the 0.2″ fibres onto a finer, square grid, the errors become correlated between neighbouring points, ie. you can’t sum over 4 pixels and get twice the S/N any more. Formally, a covariance cube is required to track this properly, but that’s rather a sledgehammer to crack a walnut. As a practical alternative, one can choose between preserving the variance in a single pixel or overestimating the variance in a pixel and preserving the overall noise statistics when summing over the dataset, which is what I’ve done. Does anyone have a good argument for doing this differently?

    Also, the fl_flux option I’ve added produces units of flux/arcsec^2 (eg. erg/cm2/s/A/arcsec2, assuming the spectra were flux calibrated) in place of simply resampling the flux density (erg/cm2/s/A/fibre) as previously. Does anyone have reason to care about a currently-nonexistent option to produce flux per (0.1″) output pixel, which I suppose would more naturally sum over spatial regions (but you could say the same thing about pixels vs A spectrally)?

    Thanks,

    James.

    0
    0
    #705

    jturner
    Keymaster

    Good news… I’ve finally had the opportunity to merge 3/4 of the ifudrgmos improvements posted here into the official gemini package code, with help pages :-). Those changes to 12 scripts are therefore likely to make it into the next public Gemini IRAF (non-commissioning) release for the GMOS-S Hamamatsu detectors, alongside support for correcting QE as a function of wavelength, which was the main feature still missing WRT Bryan’s ifuproc.

    The main ifudrgmos features not yet in the official code base are the improved flat-field-normalization (gfresponse) algorithm and the ability to subtract sky from each slit separately in gfskysub. Each of those scripts has an option (twilights and interactive mode, respectively) that’s probably broken in my version and needs a bit more work to re-enable for general use.

    As with the script versions posted here, most new steps (eg. scattered light subtraction) are currently not included in gfreduce, so if you use gfreduce you’ll have to run it more than once with different flags in order to insert those steps in between. I intend to post an example before long.

    I’ll probably re-base ifudrgmos on the official package once the latter is stable, adding back in the few remaining improvements to get a version that’s similar to r114 above but with the new help pages, QE correction, Hamamatsu support and assorted minor improvements from the past year’s official development. I hope to get a chance to merge my last changes into the official version at some point but am out of time right now.

    Cheers,

    James.

    PS. Thanks to Gemini for the time to merge this stuff and Mark Simpson for general help.

    0
    0
    • This reply was modified 2 years, 8 months ago by  jturner.
    • This reply was modified 2 years, 8 months ago by  jturner.
    #797

    jturner
    Keymaster

    I was able to merge all our ifudrgmos improvements into the recent v1.13 release of the official Gemini IRAF package, except for my flat fielding algorithm (gfresponse) & one non-essential utility.

    Here’s a new ifudrgmos version that’s re-based in turn on the v1.13 release (which includes Hamamatsu CCD support & “QE correction”), with my gfresponse (etc.) added back in. You may or may not see much difference WRT v1.13, depending on your GMOS setup & science goals.

    The only remaining API difference between ifudrgmos and the official package is the former’s “gfresponse.wavtraname” parameter, so the data reduction example I will upload will now also work with the official package with minimal changes.

    Cheers,

    James.

    Update: these attachments are old; see below for a newer version.

    1
    0
    • This reply was modified 2 years, 4 months ago by  jturner. Reason: Re-upload tarball with help database built
    • This reply was modified 8 months, 3 weeks ago by  jturner.
Viewing 15 posts - 1 through 15 (of 25 total)

You must be logged in to reply to this topic.