bug in GMOS Multi-Object Spectra PyRAF tutorial script

Home Forums Gemini Data Reduction bug in GMOS Multi-Object Spectra PyRAF tutorial script

This topic contains 3 replies, has 3 voices, and was last updated by csimpson csimpson 1 year, 1 month ago.

Viewing 4 posts - 1 through 4 (of 4 total)
  • Author
    Posts
  • #1279

    rarnason
    Participant

    Hi all,

    I am a new Gemini user attempting to understand the reduction process for GMOS data in MOS mode so that I can apply it to some of my data. I’ve been following the PyRAF tutorial script (gmos_mos_proc.py on the GMOS Cookbook) so that I can use it as a framework for reducing my own observations. I’ve been able to successfully make an observation log in sqlite3 using the provided script, however the reduction script fails for me out of the box.

    I’m using python in spyder as part of astroconda (2.7.12) but I experience an identical error using python in the Ureka package (2.7.5). The script fails during the full flat normalization process, but also returns error messages during the centerspec flat normalization process. It seems that it cannot locate the bias mastercals “MCbiasCenSp” and “MCbiasFull” that it creates in the previous steps, leading to a failure when it performs GSFLAT for the flat normalization. Is this a problem with these master cals being blank/void (i.e., not being able to get all of the files it needs when it makes the master cals) or is it a problem with the actual flat normalization process itself? I’ve attached the log of the output generated by gmos_mos_proc.py when it failed.

    As per the tutorial’s instructions, I’ve downloaded the science data and associated calibrations, and placed them all in a subfolder of my working directory marked /raw. My working directory has the masks, fileselect.py, and the reduction script gmos_mos_proc.py. I would appreciate any help with determining the source of the problem and how to fix it!

    
    --Creating GCAL Spectral Flat-Field MasterCals--
      -CenterSpec GCAL-flat normalization, non-interactive-
    GPREPARE: Using MDF defined in the header 1.0arcsec
    ERROR - GIREDUCE: Can not find bias frame: MCbiasCenSp
    WARNING - GIREDUCE: Bad Pixel Mask filename is an empty string
                        Only saturated pixels will be flagged
    ERROR - GSREDUCE: There was an apparent fatal error with GIREDUCE
     
    ERROR - GSREDUCE: Program execution failed with 1 errors
    ERROR - there was a problem running GSREDUCE.  Stopping now.
     
    ERROR - GSFLAT: Program execution failed with 1 errors.
    GPREPARE: Using MDF defined in the header 1.0arcsec
    ERROR - GIREDUCE: Can not find bias frame: MCbiasCenSp
    WARNING - GIREDUCE: Bad Pixel Mask filename is an empty string
                        Only saturated pixels will be flagged
    ERROR - GSREDUCE: There was an apparent fatal error with GIREDUCE
     
    ERROR - GSREDUCE: Program execution failed with 1 errors
    ERROR - there was a problem running GSREDUCE.  Stopping now.
     
    ERROR - GSFLAT: Program execution failed with 1 errors.
    GPREPARE: Using MDF defined in the header 1.0arcsec
    ERROR - GIREDUCE: Can not find bias frame: MCbiasCenSp
    WARNING - GIREDUCE: Bad Pixel Mask filename is an empty string
                        Only saturated pixels will be flagged
    ERROR - GSREDUCE: There was an apparent fatal error with GIREDUCE
     
    ERROR - GSREDUCE: Program execution failed with 1 errors
    ERROR - there was a problem running GSREDUCE.  Stopping now.
     
    ERROR - GSFLAT: Program execution failed with 1 errors.
    GPREPARE: Using MDF defined in the header 1.0arcsec
    ERROR - GIREDUCE: Can not find bias frame: MCbiasCenSp
    WARNING - GIREDUCE: Bad Pixel Mask filename is an empty string
                        Only saturated pixels will be flagged
    ERROR - GSREDUCE: There was an apparent fatal error with GIREDUCE
     
    ERROR - GSREDUCE: Program execution failed with 1 errors
    ERROR - there was a problem running GSREDUCE.  Stopping now.
     
    ERROR - GSFLAT: Program execution failed with 1 errors.
    GPREPARE: Using MDF defined in the header 1.0arcsec
    ERROR - GIREDUCE: Can not find bias frame: MCbiasCenSp
    WARNING - GIREDUCE: Bad Pixel Mask filename is an empty string
                        Only saturated pixels will be flagged
    ERROR - GSREDUCE: There was an apparent fatal error with GIREDUCE
     
    ERROR - GSREDUCE: Program execution failed with 1 errors
    ERROR - there was a problem running GSREDUCE.  Stopping now.
     
    ERROR - GSFLAT: Program execution failed with 1 errors.
      -Full Flat (GCAL & Twi) normalization, non-interactive-
    GPREPARE: Using MDF defined in the header GN2011BC003-01
    ERROR - GIREDUCE: Can not find bias frame: MCbiasFull
    WARNING - GIREDUCE: Bad Pixel Mask filename is an empty string
                        Only saturated pixels will be flagged
    ERROR - GSREDUCE: There was an apparent fatal error with GIREDUCE
     
    ERROR - GSREDUCE: Program execution failed with 1 errors
    ERROR - there was a problem running GSREDUCE.  Stopping now.
     
    ERROR - GSFLAT: Program execution failed with 1 errors.
    GPREPARE: Using MDF defined in the header GN2011BC003-01
    ERROR - GIREDUCE: Can not find bias frame: MCbiasFull
    WARNING - GIREDUCE: Bad Pixel Mask filename is an empty string
                        Only saturated pixels will be flagged
    ERROR - GSREDUCE: There was an apparent fatal error with GIREDUCE
     
    ERROR - GSREDUCE: Program execution failed with 1 errors
    ERROR - there was a problem running GSREDUCE.  Stopping now.
     
    ERROR - GSFLAT: Program execution failed with 1 errors.
    Traceback (most recent call last):
    
      File "<ipython-input-1-44ccb7240203>", line 1, in <module>
        runfile('/Users/rarnason/Gemini_Tutorials/GMOS_MOS/gmos_mos_proc.py', wdir='/Users/rarnason/Gemini_Tutorials/GMOS_MOS')
    
      File "/Users/rarnason/anaconda/lib/python2.7/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 714, in runfile
        execfile(filename, namespace)
    
      File "/Users/rarnason/anaconda/lib/python2.7/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 81, in execfile
        builtins.execfile(filename, *where)
    
      File "/Users/rarnason/Gemini_Tutorials/GMOS_MOS/gmos_mos_proc.py", line 318, in <module>
        gmos_mos_proc()
    
      File "/Users/rarnason/Gemini_Tutorials/GMOS_MOS/gmos_mos_proc.py", line 121, in gmos_mos_proc
        flatFull = fs.fileListQuery(dbFile, fs.createQuery(ft, qdf), qdf)
    
      File "./fileSelect.py", line 99, in createQuery
        return SQL_TYPES[listType] + dateQuerySegment(queryDict)
    
    KeyError: 'twiSpecFlat'
    

    Robin

    0
    0
    #1285
    achene
    achene
    Keymaster

    Hi!

    Looks like the script is not dealing well with your directory paths. A priori, nothing shows that IRAF is misbehaving.

    I did use the cookbook for extracting GMOS spectra before, and many edits were needed to make the script to work, as it is not like a “pipeline”. Is there a chance you can find ways to fix how the script looks for your data?

    0
    0
    #1304

    rarnason
    Participant

    Hello,

    After doing some careful troubleshooting I was able to determine the source of the error – including pre-prepared _BIAS.fits and _stack_bias.fits files causes the scripts to execute IRAF tasks incorrectly. I do have another question. I am experiencing a bug related to gbias when I attempt to apply the command to my own data and make a MasterCal bias file:

    
    SQL = fs.createQuery('bias', qdf_1_520)
    biasFull = fs.fileListQuery(dbFile, SQL, qdf_1_520)
    gmos.gbias(','.join(str(x) for x in biasFull), 'MCbiasFull.fits', **biasFlags)
    

    This code produces an error if I run it by having my dictionary “qdf_1_520” have a date range of about two months around my target observation – this generates a list (in biasFull) of about ~130 bias files to be handled by gbias. The error I get is:

    
    stsci.tools.irafglobals.IrafError: Error running IRAF task files
    IRAF task terminated abnormally
    ERROR (603, "Parameter not a legal boolean (try 'yes' or 'no') (sort)")
    

    If I narrow down the date range to fewer files (30, which is less than the 65 used by the tutorial script which operates correctly), then my code executes fine and produces a bias mastercal. Is there a silent limit on the number of bias files that can be passed to gbias (or similar tasks)?

    Cheers,

    Robin

    0
    0
    #1308
    csimpson
    csimpson
    Participant

    This appears to be due to a line-limit length in IRAF. The error described occurs if the input to a task is 1023 or more characters (a limit not normally encountered). A suitable workaround appears to be to write the list of files to a file and use IRAF’s “@” notation. So replace this line:

    gmos.gbias(','.join(str(x) for x in biasFull), 'MCbiasFull.fits', **biasFlags)

    with these:

    with open('biases.lis', 'w') as f:
        [f.write(x+'\n') for x in biasFull]
    gmos.gbias('@biases.lis', 'MCbiasFull.fits', **biasFlags)
    0
    0
Viewing 4 posts - 1 through 4 (of 4 total)

You must be logged in to reply to this topic.