5i'\ddlmZddlZddlmZddlmZmZmZddl m Z  Gdde Z y) ) defaultdictN)merge)err_exitDXError DXCLIError) basestringcjeZdZdZdZed dZed dZed dZdZ dZ dZ d Z d Z y )SystemRequirementsDictz A class representing system requirements that can be passed as "systemRequirements" or an entry of the "systemRequirementsByExecutable" to the class-xxxx/run or job/new API call (after converting it to a dictionary with as_dict()). cr|t|ts tdtj||_y)aS Example of the entrypoints input: {"main": {"instanceType": "mem2_hdd2_x2"}, "other_function": {"instanceType": "mem2_hdd2_x1", "clusterSpec": {"type": "spark", "version": "2.4.0", "initialInstanceCount": 2}}} Nz)Expected entrypoints to be a dict or None) isinstancedictrcopydeepcopy entrypoints)selfrs z/home/marpiech/ifpan-abm-pgxpred/analysis/marpiech-gwas-test/venv/lib/python3.12/site-packages/dxpy/system_requirements.py__init__zSystemRequirementsDict.__init__s0  ":k4+HEF F==5c ` ||dSt|tst|tr||ddt|iiiSt|tr7||j Dcic]\}}|ddt|iic}}St cc}}w#t $rt dYywxYw)ab Returns a SystemRequirementsDict that can be passed as a "systemRequirements" input or an entry of the "systemRequirementsByExecutable" mapping to job/new or run/ API calls. The instance_count_arg should be either a: * string or int eg. "6" or 8 * dictionary, eg. {"main": 4, "other_function": 2} N clusterSpecinitialInstanceCountzCExpected instance_count field to be either an int, string or a dict)r r intritems ValueErrorr)clsinstance_count_arg entrypointkvs rfrom_instance_countz*SystemRequirementsDict.from_instance_count(s [!)4y ,j9ZHZ\_=`J9OQTUgQh8i(jkll,d3]o]u]u]wxUYUVXYA 0FA/OPPxyy y [ Y Z [s. B6B%B(B B BB-,B-c ||dSt|tr ||d|iiSt|tr,||jDcic] \}}|d|i c}}St dcc}}w)aw Returns SystemRequirementsDict that can be passed as a "systemRequirements" input or an entry of the "systemRequirementsByExecutable" mapping to job/new or run/ API calls. The instance_type_arg should be either a: * string, eg. mem1_ssd1_x2 * dictionary, eg. {"main": "mem2_hdd2_x2", "other_function": "mem2_hdd2_x1"} N instanceTypez*+ +% 288: :OJ|/25z *%e, :4 ?##rc dtj|j}t|j Dcgc]\}}|j d||fc}}}t|j Dcgc]%\}}||jvsd|jvr||f'c}}}|j D]R\}}||dj |jj ||jj ddT|jj D]o\}}||vs d|jvsd|jdvs,dtj|jddi||<||dj |dqt|Scc}}wcc}}w)a Returns SystemRequirementsDict can be passed in a "systemRequirements" input or as an entry of the "systemRequirementsByExecutable" mapping to app-xxx/run, e.g. {'fn': {'clusterSpec': {initialInstanceCount: 3, version: "2.4.0", ..}}} Since full clusterSpec must be passed to the API server, we need to retrieve the cluster spec defined in app doc's systemRequirements and overwrite the field initialInstanceCount with the value the user passed to dx run for each entrypoint. initialInstanceCount is currently the only clusterSpec's field the user is allowed to change at runtime. A few scenarios when requesting instance count for different entrypoints with dx run and the resulting merged systemRequirements (merged_cluster_spec). The bootstapScript field here is only one of many (version, ports, etc) that should be copied from app spec to merged_cluster_spec: Requested: {"*": 5} App doc: {"main": "clusterSpec": {"initialInstanceCount": 7, bootstrapScript: "x.sh"}, "other": "clusterSpec": {"initialInstanceCount": 9, bootstrapScript: "y.sh"}} Merged: {"main": "clusterSpec": {"initialInstanceCount": 5, bootstrapScript: "x.sh"}, "other": "clusterSpec": {"initialInstanceCount": 5, bootstrapScript: "y.sh"}} Requested: {"*": 15} App doc: {"main": "clusterSpec": {"initialInstanceCount": 7, bootstrapScript: "x.sh"}, "other": "clusterSpec": {"initialInstanceCount": 9, bootstrapScript: "y.sh"}, "*": "clusterSpec": {"initialInstanceCount": 11, bootstrapScript: "y.sh"}} Merged: {"main": "clusterSpec": {"initialInstanceCount": 15, bootstrapScript: "x.sh"}, "other": "clusterSpec": {"initialInstanceCount": 15, bootstrapScript: "y.sh"}, "*": "clusterSpec": {"initialInstanceCount": 15, bootstrapScript: "y.sh"}} Requested: {"main": 12} App doc: {"main": "clusterSpec": {"initialInstanceCount": 7, bootstrapScript: "x.sh"}, "other": "clusterSpec": {"initialInstanceCount": 9, bootstrapScript: "y.sh"}} Merged: {"main": "clusterSpec": {"initialInstanceCount": 12, bootstrapScript: "x.sh"}} Requested: {"main": 33} App doc: {"*": "clusterSpec": {"initialInstanceCount": 2, bootstrapScript: "z.sh"}} Merged: {"main": "clusterSpec": {"initialInstanceCount": 33, bootstrapScript: "z.sh"}} Requested: {"main": 22, "*": 11} App doc: {"*": "clusterSpec": {"initialInstanceCount": 2, bootstrapScript: "t.sh"}} Merged: {"main": "clusterSpec": {"initialInstanceCount": 22, bootstrapScript: "t.sh"}, "*": "clusterSpec": {"initialInstanceCount": 11, bootstrapScript: "t.sh"}} r*)rrrrrgetupdater )rsrdmerged_cluster_specrr entry_ptr2s roverride_cluster_specz,SystemRequirementsDict.override_cluster_specesX#mmD,<,<=#7J7P7P7R#wtq!VWV[V[\iVjVvQF#wx#7J7P7P7R$N>N7NS`dhdtdtuxdySy1> dN^N^_bNcdqNr@s0t#H-#H-m<CCC DVW X &&9::)$x$^.B\B\B`B`adBe^imiyiyi}i}~jA^EIEUEUEYEYZ]E^KN ^%k22rc  ||y|xsi}|xsi}i}tt|jt|jzD]1}t|j |ifi|j |i||<3|S)a= Merges the values of two dictionaries, which are expected to be dictionaries, e.g d1 = {'a': {'x': pqr}} d2 = {'a': {'y': lmn}, 'b': {'y': rst}} will return: {'a': {'x': pqr, 'y': lmn}, 'b': {'y': rst}}. Collisions of the keys of the sub-dictionaries are not checked. N)setlistr>rr6)rd1d2addedkeys r_add_dict_valuesz'SystemRequirementsDict._add_dict_valuess :"* X2 X2tBGGIbggi89 DCbffS"oC"&&b/CE#J D rct|ts td|j|j|j}t|S)Nz0Developer error: SystemRequirementsDict expected)r r rrKr)rotheradded_entrypointss r__add__zSystemRequirementsDict.__add__sD%!78LM M 11$2B2BEDUDUV%&788rc|jS)N)r)rs rr=zSystemRequirementsDict.as_dictsrN)r5)r))__name__ __module__ __qualname____doc__r classmethodr!r'r3r;rCrKrOr=rrr r sg 6[[(VV&$$&C;J3(9  rr ) collectionsrrutilsr exceptionsrrrcompatr objectr rVrrr\s*# 77~ V~ r