5i;=dZddlmZmZmZmZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlmZddlmZddZddZd Zd Zd Zd Zd ZdZdZdZdZdZddZy)a This module provides support for file download and upload. It calculates the location of the input and output directories. It also has a utility for parsing the job input file ('job_input.json'). We use the following shorthands == input directory $HOME/in == output directory $HOME/out A simple example of the job input, when run locally, is: { "seq2": { "$dnanexus_link": { "project": "project-1111", "id": "file-1111" } }, "seq1": { "$dnanexus_link": { "project": "project-2222", "id": "file-2222" } } "blast_args": "", "evalue": 0.01 } The first two elements are files {seq1, seq2}, the other elements are {blast_args, evalue}. The files for seq1,seq2 should be saved into: /seq1/ /seq2/ An example for a shell command that would create these arguments is: $ dx run coolapp -iseq1=NC_000868.fasta -iseq2=NC_001422.fasta It would run an app named "coolapp", with file arguments for seq1 and seq2. Both NC_* files should be the names of files in a DNAnexus project (and should be resolved to their file IDs by dx). Subsequently, after dx-download-all-inputs is run, file seq1 should appear in the execution environment at path: /seq1/NC_000868.fasta File Arrays { "reads": [{ "$dnanexus_link": { "project": "project-3333", "id": "file-3333" } }, { "$dnanexus_link": { "project": "project-4444", "id": "file-4444" } }] } This is a file array with two files. Running a command like this: $ dx run coolapp -ireads=A.fastq -ireads=B.fasta will download into the execution environment: /reads/A.fastq B.fastq )print_functionunicode_literalsdivisionabsolute_importN) basestring)DXErrorc||}ntjjd}tjj |d}|S)z :param job_homedir: explicit value for home directory, used for testing purposes :rtype: string :returns: path to input directory Returns the input directory, where all inputs are downloaded HOMEinosenvirongetpathjoin) job_homedirhome_diridirs |/home/marpiech/ifpan-abm-pgxpred/analysis/marpiech-gwas-test/venv/lib/python3.12/site-packages/dxpy/utils/file_load_utils.py get_input_dirrbs:::>>&) 77<<$ 'D Kc||}ntjjd}tjj |d}|S)z :param job_homedir: explicit value for home directory, used for testing purposes :rtype: string :returns: path to output directory Returns the output directory, where all outputs are created, and uploaded from r outr )rrodirs rget_output_dirrrs:::>>&) 77<<% (D Krctjjd}tjj |dS)z> :rtype: string :returns: path to input JSON file r zjob_input.jsonr rs rget_input_json_filers, zz~~f%H 77<<"2 33rctjjd}tjj |dS)z? :rtype: string :returns: Path to output JSON file r zjob_output.jsonr rs rget_output_json_filer!s, zz~~f%H 77<<"3 44rct} tj|y#t$r)}|jtj k(rnYd}~yd}~wwxYw)z| Warning: this is not for casual use. It erases the output json file, and should be used for testing purposes only. N)r!rremoveOSErrorerrnoENOENT)res rrm_output_json_filer(sG !D $  77ell "   s" AAActjj|stj|ytjj |rt d|zy)zl :param path: path to directory to be created Create a directory if it does not already exist. z9Path %s already exists, and it is a file, not a directoryN)rrexistsmkdirisfile Exception)rs r ensure_dirr.sH 77>>$   77>>$ WZ^^_ _ rcjddg}||vrtdj||jddS)a :param fname: the basename of a file (e.g., xxx in /zzz/yyy/xxx). :returns: a valid unix filename :rtype: string :raises: DXError if the filename is invalid on a Unix system The problem being solved here is that *fname* is a python string, it may contain characters that are invalid for a file name. We replace all the slashes with %2F. Another issue, is that the user may choose an invalid name. Since we focus on Unix systems, the only possibilies are "." and "..". .z..zInvalid filename {}/z%2F)r formatreplace)fname bad_filenamess rmake_unix_filenamer6s=$KM +2259:: ==e $$rcrt|jDcic] \}}||vs ||c}}Scc}}wN)listitems)dict_ excl_keyskvs r filter_dictr?s0!%++-0 GTQAY4FAqD GG Gs 33cr fd}|}tjt g fd fd}t|jD])\}}t |tr |||  |d|+t|jDcic] \}}| vs ||}}} |fScc}}w)agExtract list of files, returns a set of directories to create, and a set of files, with sources and destinations. The paths created are relative to the input directory. Note: we go through file names inside arrays, and create a separate subdirectory for each. This avoids clobbering files when duplicate filenames appear in an array. ctt5}tj|}|cdddS#1swYyxYwr8)openjsonload)fh job_inputjob_input_files rget_input_hashz/get_job_input_filenames..get_input_hashs2 . ! R " I   s.7ctj|sytj|}t|tjsyt |j }|}| tjj||}|jtjj||||jdj|y)N) trg_fnamehandler src_file_id) dxpy is_dxlink get_handler isinstanceDXFiler6namerrrappendid)inamesubdirvaluerKfilenametrg_dirdirsfiless radd_filez)get_job_input_filenames..add_files~~e$ ""5)'4;;/ %gll3  ggll7F3G e "'',,w*I(/,3JJ8 9 Grct|}|dk(rytt|dz }j|t|D])\}}t|j |}|||+y)Nr)lenstrrS enumeratezfill) input_namelinks num_files num_digitsilinkrVr\rZs radd_file_arrayz/get_job_input_filenames..add_file_arrayskJ > Y]+,  J ' /GAtV\\*-F Z . /rN) collections defaultdictr9r:rP) rGrHrFrircrWkeyval rest_hashr\rZr[s ` @@@rget_job_input_filenamesros I  # #D )E D ,/")//"34. E eT " :u - Zu - .+/y/@*AVhc3SPUEUcVIV  !!Ws  B3&B3cbd}dtjvritjtj}|ddk(rtj|j d|j d}d|vre|d}n_dtjvrMtjd}t |5}tj|}|j d}ddd|iSd }i}|D]8}|d } d |vs g|| <|d D]} || s || j| !:|S#1swYQxYw) z Extract the inputSpec patterns, if they exist -- modifed from dx-upload-all-outputs Returns a dict of all patterns, with keys equal to the respective input parameter names. N DX_JOB_IDfunctionmainappapplet inputSpecDX_TEST_DXAPP_JSONc d|vS)N*)patterns ris_legal_patternz1get_input_spec_patterns..is_legal_pattern/s g~rrRpatterns) rrrMdescribeJOB_IDrrBrCrDrS) input_specjob_descdescpath_to_dxapp_jsonfd dxapp_jsonr| patterns_dictspecrRps rget_input_spec_patternsrs7 Jbjj ==- J 6 )==eX\\(5K!LMDd"!+.  +ZZ(<= $ % 52J# 4J 5  M2F|  "$M$ *% 2#A&!$'..q1 2 2 + 5 5s .'D%%D.cH||S||St|t|kr|S|Sr8)r_)rqs rchoose_shorter_stringr=s/yy 1vA Hrc$t|\}}}tfd}d}tj|}t |}t |j D]\} } | D]} | d} tjj| } || | }|| }|dj| d|dj| |dj||djtjj|| ||fS)aO This function examines the input file, and calculates variables to instantiate in the shell environment. It is called right before starting the execution of an app in a worker. For each input key, we want to have $var $var_filename $var_prefix remove last dot (+gz), and/or remove patterns $var_path $HOME/in/var/$var_filename For example, $HOME/in/genes/A.txt B.txt export genes=('{"$dnanexus_link": "file-xxxx"}' '{"$dnanexus_link": "file-yyyy"}') export genes_filename=("A.txt" "B.txt") export genes_prefix=("A" "B") export genes_path=("$HOME/in/genes/A.txt" "$HOME/in/genes/B.txt") If there are patterns defined in the input spec, then the prefix respects them. Here are several examples, where the patterns are: *.bam, *.bwa-index.tar.gz, foo*.sam, z*ra.sam file name prefix matches foo.zed.bam foo.zed *.bam xxx.bwa-index.tar.gz xxx *.bwa-index.tar.gz food.sam food foo*.sam zebra.sam zebra z*ra.sam xx.c xx xx.c.gz xx The only patterns we recognize are of the form x*.y. For example: legal *.sam, *.c.py, foo*.sam, a*b*c.baz ignored uu.txt x???.tar mon[a-z].py c jd}j|}|L|D]G}tj||s|jd\}}}t||dt | }I||St j j|}|ddk(r"t j j|d}|dS)Nryr^z.gzr)rfnmatch rpartitionrr_rrsplitext) basenamerl best_prefixr}r{_ right_piecepartsrs r get_prefixz%analyze_bash_vars..get_prefixrs  $$S)  # c??8W5(/(:(:3(?%Aq+"7 XN`PST_P`O`Ea"bK c  " GG$$X.EQx5 ((q28OrcggggdS)N)rKrprefixrrzrzrrfactoryz"analyze_bash_vars..factorys2"bIIrrJrKrrr) rorrjrkrr9r:rrrrSr)rGrr file_entriesrnrrfile_key_descs rel_home_dirrlentriesentryrXrrk_descrs @ranalyze_bash_varsrGsN"9!HA|Y+-M"J ,,W5N -L\//12 H W HE[)Hww''1H#.F#C(F 9  $ $U9%5 6 :  % %h / 8  # #F + 6N ! !"'',,|X"F G H H 9 $$rc t||\}}d fd}i  fd}t|jD]\}}||||t|jD]V\} }|| ||d|| dz||d|| dz||d|| d z||d X S) a :param job_input_file: path to a JSON file describing the job inputs :param job_homedir: path to home directory, used for testing purposes :param check_name_collision: should we check for name collisions? :return: list of lines :rtype: list of strings Calculates a line for each shell variable to instantiate. If *check_name_collision* is true, then detect and warn about collisions with essential environment variables. cd}t|tr|}nXt|tjr)t j tj |}nt j |}tj|Sr8) rPrrMrQrCdumpsdxlinkshlexquote)elemresults rstring_of_elemz%gen_bash_vars..string_of_elemsX dJ 'F dkk *ZZ D 12FZZ%F{{6""rct|tr6dj|Dcgc] }| c}}dj|S|Scc}w)N z( {} ))rPr9rr2)rmvitemstringrs rstring_of_valuez&gen_bash_vars..string_of_valuesK c4 XX#F~e4FGF??6* *!#& &GsAcru|tjvr |vr||<ytjj t j jjdj|dzy||<y)zW In the absence of a name collision, create a line describing a bash variable. z?Creating environment variable ({}) would cause a name collision N) rrsysstderrwriterMutilsprintingfillr2)rlrmcheck_name_collision var_defs_hashs r gen_text_line_and_name_collisionz7gen_bash_vars..gen_text_line_and_name_collisionsq "**$M)A%( c"   !4!4!9!9U\\]`a"""%M# rrK_namer_prefixr_pathr)rr9r:) rGrrrrnrrrlrfile_keyrrs ` @@r gen_bash_varsrs!2.+ NNI#'M %)//+,E T(od.CDE~3356\$(?4 ?3ST(G);_TR\M]=^_(I)=tT\~?^_(G);_TRX\=Z[ \ rr8)NT)__doc__ __future__rrrrrCrrrrrjr%rMcompatr exceptionsr rrrr!r(r.r6r?rorrrrrzrrrs{"@DSR      "45  `%(HE"P'V J%Z8r