5iEdZddlmZmZmZmZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlmZddlmZddlZddlZddlmZiZdae j4e j6ze j8ze j:zZdZd Z dd Z!dd Z"d Z#d Z$dZ%GddejLZ'GddeZ(Gdde)Z*y)zH Utilities used in the DNAnexus execution environment and test harness. )print_functionunicode_literalsdivisionabsolute_importN)Mapping)wraps)AppInternalErrorct|S)zw Returns an equivalent unicode object, trying harder to avoid dependencies on the Python default encoding. )str)os w/home/marpiech/ifpan-abm-pgxpred/analysis/marpiech-gwas-test/venv/lib/python3.12/site-packages/dxpy/utils/exec_utils.py _safe_unicoder%s q6Mct|tjr t|S|jj dzt|zS)z* Formats the specified exception. z: ) isinstancedxpyAppErrorr __class____name__)es r_format_exception_messager,s< !T]]#Q ;;  $ &q)9 99rctdz atj}tjnt j  t jjtjtjtj}nS| tjjdd}|*t!dd5}t#j$|}ddd||d}t!d d 5}|j'd d d zzdddtd|jdd|jd t)|ddi|d}|Ztj0|t!dd 5}t#j2||dt4|j'dddd|S|S#tjj$rtdYmwxYw#1swY xYw#1swYxYw#tj*$r}t-||dd}~wt.$r}t-||d}~wwxYw#1swY|SxYw)aTriggers the execution environment entry point processor. Use this function in the program entry point code: .. code-block:: python import dxpy @dxpy.entry_point('main') def hello(i): pass dxpy.run() This method may be used to invoke the program either in a production environment (inside the execution environment) or for local debugging (in the debug harness), as follows: If the environment variable *DX_JOB_ID* is set, the processor retrieves the job with that ID from the API server. The job's *function* field indicates the function name to be invoked. That function name is looked up in the table of all methods decorated with *@dxpy.entry_point('name')* in the module from which :func:`run()` was called, and the matching method is invoked (with the job's input supplied as parameters). This is the mode of operation used in the DNAnexus execution environment. .. warning:: The parameters *function_name* and *function_input* are disregarded in this mode of operation. If the environment variable *DX_JOB_ID* is not set, the function name may be given in *function_name*; if not set, it is set by the environment variable *DX_TEST_FUNCTION*. The function input may be given in *function_input*; if not set, it is set by the local file *job_input.json* which is expected to be present. The absence of *DX_JOB_ID* signals to :func:`run()` that execution is happening in the debug harness. In this mode of operation, all calls to :func:`dxpy.bindings.dxjob.new_dxjob()` (and higher level handler methods which use it) are intercepted, and :func:`run()` is invoked instead with appropriate inputs. NzATODO: FIXME: the EE client should die if logging is not availableDX_TEST_FUNCTIONmainzjob_input.jsonr)functioninputjob_error_reserved_spacewz^This file contains reserved space for writing job errors in case the filesystem becomes full. a @Invokingrwithrr) error_typezjob_output.jsonr )indentcls ) RUN_COUNTosgetcwdrJOB_IDlogging basicConfig getLogger addHandler DXLogHandler exceptionsDXErrorprintdescribeenvirongetopenjsonloadwriteENTRY_POINT_TABLEr save_error Exceptionchdirdump DXJSONEncoder) function_namefunction_inputdx_working_dirjobfhresultrs rrunrI7s`NIYY[N {{ W     * *4+<+<+> ?mmDKK(  JJNN+=vFM  !&, /!%2 /)>B (# .B" ru}AvAA BB *cggj)637773CD"3z?3Cc'lC  #S ) R &"QMB  M6ME&& W U V W / / BB ==1n< 1n%   MsZ5F: G);G6H?/I:(G&%G&)G36G?IH$$ I0 H==IIc6tjvtj| tjdt dd5}t jd|t|di||jddddyy#YPxYw#1swYyxYw)Nr zjob_error.jsonr!error)typemessager() rr-r+r@unlinkr9r:rArr<)r working_dirr%rGs rr>r>s {{   II0 1"C ( B 7ZD]^_D`$abdfg       sB6BB BcLt|tjrtj|}|St|tr(|j D]\}}t |||<|St|tr*tt|D]}t ||||<|SN) rrDXObjectdxlinkritemsconvert_handlers_to_dxlinkslistrangelen)xkeyvalueis rrUrUs!T]]# KKN H Aw '') 8JC07AcF 8 H At s1v 5A.qt4AaD 5 Hrctj}t}|jdgD]}i}|jddk(r t|d<nd|jddk(r t |d<nF|jddk(r t |d<n(|jddk7r|j|d|jddk7r |d |d <|jd |dzfi|i}t|j|jD]+\}}| ||vr tj|||<'|||<-|S#t$rsd d lm} | |d ddid \} } } t#| | | | t%| dk7rtdj'|t)j*| d d| ||<YwxYw)N inputSpecrLintfloatbooleanstringnameoptionalrequiredz--r)resolve_existing_pathFidT)ask_to_resolver6 allow_multrzValue {v} could not be resolved)v) project_id)argparseArgumentParsersetr8r_r`booladd add_argumentvars parse_argsrTr:loads ValueErrordxpy.utils.resolverrfr5rXformatrrS) argsapp_specparser json_inputsispeckwargsinputsr\r[rfprojectpathresultss rparse_args_as_job_inputrs  $ $ &F%Kk2.< 99V  % F6N YYv ' )"F6N YYv ) +!F6N YYv ( * OOE&M * 99Z D (%*:%6!6F: D5=0;F;< F**4017795 =   N JJu-q F1I M NE)>uUZfjlpeqD*E&wgtW-?c'la&7$%F%M%MPU%M%VWW KK 4(8WMq  Ns(E  A9GGcfd}|S)zUse this to decorate a DNAnexus execution environment entry point. Example: .. code-block:: python @dxpy.entry_point('main') def hello(i): pass cBt<tfd}|S)Nc|i|SrQr))rxr}fs r wrapped_fz,entry_point..wrap..wrapped_fsd%f% %r)r=r)rrentry_point_names` rwrapzentry_point..wraps+./*+ q &  &rr))rrs` r entry_pointrs KrceZdZdZdZy)rBzL Like json.JSONEncoder, but converts DXObject objects into dxlinks. ct|tjrtj|Stj j ||SrQ)rrrRrSr: JSONEncoderdefault)selfobjs rrzDXJSONEncoder.defaults8 c4== );;s# #''c22rN)r __module__ __qualname____doc__rr)rrrBrBs 3rrBc eZdZy)DXExecDependencyErrorN)rrrr)rrrrsrrcTeZdZdZdZddZdZdZdZddZ d Z d Z d Z d Z d Zy)DXExecDependencyInstallera( Installs dependencies specified by the job. Dependencies are processed in the order specified in the bundledDepends, execDepends, and dependencies arrays of the runSpec hash (the former two are deprecated). Neighboring package dependencies of the same type are grouped. )aptgemcpancranpippip3Ncd|vr tdd|dvrd|vr td||_|d|_||_|jj dd|_||_g|_tj|j|jj dg|jj d gD]}|j||j|}|d d vr|d k(r3d |vr@|j |d vr/|jdj|d |j wt|jdk(s#|jdd|k7s||j vr2|jj#|gt|jd|jddj#|y)a :param executable_desc: The description of the executable of this job, which can be obtained in the response of the /executable-x/describe request. This dict must contain the following keys: - "runSpec" :type executable_desc: dict :param job_desc: The description of this job, which can be obtained in the response of the /job-x/describe request. If ``executable_desc["runSpec"]`` has key "bundledDependsByRegion", then this dict must contain the following keys: - "region" :type job_desc: dict runSpeczAExpected field "runSpec" to be present in executable description"bundledDependsByRegionregionz(Expected key 'region' in job descriptionrr execDepends dependenciesrc)z dx-toolkitjqzpython-argcompleterstageszDSkipping dependency {} because it is inactive in stage (function) {}rrL)rLdepsindexrN)r exec_descrun_specjob_descr8stagelogger dep_groups itertoolschain_get_local_bundled_dependencies_validate_dependency_get_dependency_typelogrwrX group_pmsappend)rexecutable_descrrdepdep_types r__init__z"DXExecDependencyInstaller.__init__s O +'(kl l #y'A AhV^F^'(RS S(' 2   ]]&&z6:  ??4#G#G#I#'==#4#4]B#G#'==#4#4^R#HJ 4C  % %c *005H6{HHXY^M^34::S]#B_ffgjkqgrgkgqgqst4??#q(DOOB,?,G8,SW_gkguguWu&&"sSWSbSbOc'de OOB  ' . .s 3' 4rc|jjd}||jjdgS|jd}||S)NrbundledDependsr)rr8r)rbundled_depends_by_region job_regions rrz9DXExecDependencyInstaller._get_local_bundled_dependencies/sQ$(MM$5$56N$O! $ ,==$$%5r: :x0J,Z8 8rcj|jr|jj|yt|yrQ)rinfor5)rrMs rrzDXExecDependencyInstaller.log8s" ;; KK  W % 'Nrcd}d}d}d}|j|||}d;d}|d|d} }|d k(r|j||  S|d k(r d || d zS|dk(r d|| d zS|dk(rWg} | D]1} | jd| dzd| vs| dxxd| dzz cc<3djtt| S|dk(r d|| dzS|dk(rd} d} | dj| z } d }d!}g} | D]O}d|vrd"jdzVDXExecDependencyInstaller.generate_shellcode..make_pm_atoms..Gs/vhiQvYIYZN*;AiL*H`bcvs! )joinmapr )packagesr package_atomss ` r make_pm_atomszCDXExecDependencyInstaller.generate_shellcode..make_pm_atomsFs$vmuvM88C]34 4rrLrr)rrzpip install --upgrade z==)rrzpip3 install --upgrade rz gem install rcrrz --version z && rzcpanm --notest ~rzhttp://cloud.r-project.orgzGdie <- function() { q(status=1) }; options(error=die); options(warn=2);z?r <- getOption("repos"); r["CRAN"] = "{repo}"; options(repos=r))repozR -e '{preamble}; {cmd}'zinstall.packages("devtools")zArequire(devtools); install_version("{name}", version="{version}")zinstall.packages("{}"))preamblecmdc3$K|]}d|v yw)zrequire(devtools)Nr))rrs rrz?DXExecDependencyInstaller.generate_shellcode..gsB#&#-Bsrgitzapt-get install --yes git makez export GIT_SSH=dx-git-ssh-helperdestdirz $(mktemp -d)z mkdir -p %szcd %sz git clone urlz ([^\/]+)$rz.gitzcd '%s'tagz git checkout build_commands()z'Package manager type {pm} not supported)pm)=r))rwrrrr anyinsertr8researchgroupendswithrXr)r dep_groupbase_apt_shellcodedx_apt_update_shellcode apt_err_msgapt_shellcode_template apt_shellcoderrrcommandsrr r_preambler_cmd_template bootstrap_cmdpackagerr subcommands build_dirsubdirs rgenerate_shellcodez,DXExecDependencyInstaller.generate_shellcode>sjy#^a !_.55=T;M8C6E  5'v. &0A( u  ''-*A'B B  +mHX\.]] ]  ,}XY]/^^ ^  H A6 :;>RLMAiL$@@L A;;s312 2  $}XQT'UU U  /DbJ [bbhlbm mJ7N:MH# U'd]ddognoC299'&/JC 5 5zs 5 ST  U BBB>#8#8*R_#8#`a;;x( (  8:\]H F  > BC ""=9#<=""7Y#67""<#c%j/#AB;CJ@FFqI??6*#Mc&k\2F""9v#56C<&&SZ'HI#s*&&s3/?+@'ABfkk+&> > DE F;;x( ('(Q(X(X\d(X(ef frc6tj|d||y)NT)shellstdoutstderr) subprocess check_call)rrlog_fhs rrIzDXExecDependencyInstaller.runscfVLrc |jdj|ddjd|dD|j|}tj jt jdjd i|} t|d5}|j||dddy#1swYyxYw#tj$rn}t|5}tjj|jdddn #1swYnxYwt!d jd i|d}~wwxYw) NzInstalling {} packages {}rLz, c3&K|] }|d yw)rcNr))rrs rrz?DXExecDependencyInstaller._install_dep_group..s=gcc&k=gsrzdx_{type}_install_{index}.logr!)rz-Error while installing {type} packages {deps}r))rrwrrr+rtempfile gettempdirr9rIrCalledProcessErrorsysrr<readr)rrr log_filenamerGrs r_install_dep_groupz,DXExecDependencyInstaller._install_dep_groups! ,33If4E48II=gU^_eUf=g4gi j%%i0ww||H$7$7$9;a;Z;a;a;ndm;no  mlC( )BR( ) ) ),, ml# ,r   + , , ,'(^(W(^(^(kaj(kl l msN CB;2C;CCCE E%.D ED% !"EEc|djddjdr|jdjd i| t j |d|dtj |jdjtj|dy|jd jd i|y#tjj$rt j |d|dYwxYw) Nrgz$dnanexus_linkrzfile-zDownloading bundled file {name}rc)rz dx-unpack {}zHSkipping bundled dependency "{name}" because it does not refer to a filer)) r8 startswithrrwrdownload_dxfile WORKSPACE_IDr3ResourceNotFoundrIshlexquote)rbundles r_install_dep_bundlez-DXExecDependencyInstaller._install_dep_bundles $<  ,b 1 < 4K\K\] HH^**5;;vf~+FG H HHf_ffpiop q ??33 C$$VD\6&>B Cs,C 9DDc|jD]3}|ddk(r|j|dd#|j|5y)NrLr rr)rr r)rrs rinstallz!DXExecDependencyInstaller.installsH 3I H,((6):1)=>'' 2  3rcd|vrtdj||jddk(rd|vrtdj|yy)Nrcz@Expected field "name" to be present in execution dependency "{}"package_managerrrz5Execution dependency "{}" does not have a "url" field)rrwr8rrs rrz.DXExecDependencyInstaller._validate_dependencys\  '(j(q(qru(vw w WW& '5 0U#5E'(_(f(fgj(kl l6F 0rc0d|vry|jddS)Nrgr rr)r8rs rrz.DXExecDependencyInstaller._get_dependency_types 3;77,e4 4rrQ)rrrrrrrrrrIrr rrrr)rrrrsG>I04d9 ?gBM m r3m 5rr)NN)+r __future__rrrrr+rr:rr.rlrbrrrcollections.abcr functoolsrr rr3r r=r* ascii_lettersdigits punctuation whitespaceASCII_PRINTABLErrrIr>rUrrrrBrobjectrr)rrrs"SRTTTTT# ) &&69K9KKfN_N__:[z   $L(3D$$3 , z5z5r