Enfortumab Vedotin With or Without Pembrolizumab in Rare Genitourinary Tumors (E-VIRTUE)

* INCLUSION CRITERIA: * Participants must have histologically confirmed locally advanced or metastatic pure adenocarcinoma of the urinary tract, pure squamous cell carcinoma of the urinary tract, or treatment-refractory testicular germ cell tumors. Note: For the purposes of enrollment, the urinary tract is defined as the renal pelvis, ureter, bladder, and urethra. * Participants must have measurable disease, per RECIST 1.1. * Participants must have locally advanced or metastatic disease defined as new or progressive lesions on cross sectional imaging. * Participants in Cohorts A1, B1, and C1 must have received prior anti-PD-1/PD-L1 therapy in any setting. * Participants in Cohorts A2, B2, and C2 must be immune checkpoint inhibitor naive. * Participants may be systemic treatment naive except for participants with testicular germ cell tumors, who must have received and be refractory to all standard options of curative-intent treatment. * Age \>= 18 years. * Eastern Cooperative Oncology Group (ECOG) performance status \<= 1 (Karnofsky \>= 70%) * Participants must have adequate organ and marrow function as defined below: * Hemoglobin (Hgb) \>= 9g/dL * Absolute neutrophil count (ANC) \>= 1,500/mcL * Platelets \>= 100,000/mcL * Total bilirubin \<= 1.5 x upper limit of normal (ULN) (\<= 3 x ULN in participants with known/suspected Gilbert's disease) * Aspartate aminotransferase (AST)/Alanine aminotransferase (ALT) \<= 2.5 x institutional ULN * Creatinine clearance (CrCl) \>= 30 mL/min/1.73m\^2 as estimated per institution standards. * Pre-study treatment tissue availability (sufficient tissue for 25 unstained slides) is mandatory for enrollment. If tissue is determined to be insufficient/unsuitable, a fresh biopsy prior to study therapy will be required. * Human immunodeficiency virus (HIV) positive participants are eligible if on stable dose of highly active antiretroviral therapy (HAART) for at least 3 months, CD4 counts are \> 200 cells/mm\^3 and viral load (VL) is undetectable. * Hepatitis B virus (HBV) positive participants are eligible if they have been treated or are on an appropriate course of antivirals at study entry and with planned monitoring and management according to appropriate guidance. For previously treated participants or those with prior infection that has been cleared, prophylaxis is permitted, and hepatology consultation recommended. * Hepatitis C virus (HCV) positive participants are eligible if participants are on active HCV therapy at study entry and on a stable dose of antivirals without documented clinically significant impaired liver function test or hematologic abnormalities and with planned monitoring and management according to appropriate labeling, or if they are post-treatment for HCV. Participants that are positive for hepatitis C must have a negative polymerase chain reaction (PCR). * Women of child-bearing potential (WOCBP) must agree to use an effective method of contraception (barrier, surgical sterilization, abstinence prior to study entry, for the duration of study participation, and for at least 4 months after the last dose of study drug(s). * Men able to father children must agree to use an effective method of contraception (barrier, surgical sterilization, abstinence) for the duration of the study treatment and up to 4 months after the last dose of the study drug(s). We also will recommend men with female partners of childbearing potential to ask female partners to be on an effective birth control (hormonal, intrauterine device (IUD), surgical sterilization). * Breastfeeding participants must be willing to discontinue breastfeeding from study treatment initiation through 4 months after the last dose of the pembrolizumab and 3 weeks after the last dose of EV. * Participants must be able to understand and willing to sign a written informed consent document. EXCLUSION CRITERIA: * Participants with prior investigational drug, chemotherapy, immunotherapy, or any prior radiotherapy (except for palliative bone directed therapy) within the past 2 weeks prior to the first study drug administration. Note: FDA-approved hormonal therapy for the treatment or prevention of other malignancies (e.g., breast cancer, prostate cancer) are allowed to be continued where in the opinion of the investigator stopping such therapies may increase the risk of disease progression. Potential drug-drug interactions with the hormonal agent will be assessed by the investigator prior to enrollment. * Participants with prior treatment with EV or other MMAE-based ADCs. * Participants with preexisting sensory or motor neuropathy Grade \>= 2. * History of severe allergic reactions attributed to compounds of similar chemical or biologic composition to EV and/or pembrolizumab. * Symptomatic or untreated central nervous system (CNS) metastases. Note: Participants with previously treated brain or CNS metastases are eligible if the participants have recovered from any acute effects of radiotherapy and not requiring steroids, and any whole brain radiation therapy or any stereotactic radiosurgery was completed at least 2 weeks prior to initiation of study therapy. * Participants will be excluded if they have an active autoimmune disease that might deteriorate when receiving pembrolizumab except for: * Diabetes type I, eczema, vitiligo, psoriasis, or hypo- or hyperthyroid diseases not requiring immunosuppressive treatment. * Participants requiring hormone replacement with corticosteroids are eligible if the steroids are administered only for the purpose of hormonal replacement and at doses \<= 10 mg of prednisone or equivalent per day. * Administration of steroids for other conditions through a route known to result in a minimal systemic exposure (topical, intranasal, intro-ocular, or inhalation). * Participants on systemic intravenous or oral corticosteroid therapy with the exception of physiologic doses of corticosteroids (\<= the equivalent of prednisone 10 mg/day) or other immunosuppressive agents such as azathioprine or cyclosporin A are excluded on the basis of potential immune suppression. For these participants these excluded treatments must be discontinued at least 1 week prior to treatment initiation for recent short course use (\<= 14 days) or discontinued at least 4 weeks prior to treatment initiation for long term use (\> 14 days). In addition, the use of corticosteroids as premedication for contrast-enhanced studies is allowed prior to treatment initiation and on study. * History of uncontrolled diabetes mellitus within 3 months before the first dose of EV. Uncontrolled diabetes is defined as hemoglobin A1C (HbA1c) \>= 8% or HbA1c between 7 and \< 8% with associated diabetes symptoms (polyuria or polydipsia) that are not otherwise explained. * Active keratitis or corneal ulcerations. * Participants who have received or will receive a live vaccine within 30 days prior to the first administration of study intervention. Seasonal flu vaccines that do not contain a live virus are permitted. Locally approved COVID-19 vaccines are permitted. * Pregnant women as evaluated by a positive serum or urine beta-human chorionic gonadotropin (Beta-hCG) test at screening. * Participants with severe uncontrolled intercurrent illness that would limit compliance with study requirements, as evaluated by history, physical exam, and chemistry panel. * Participants with severe uncontrolled intercurrent illness that would limit compliance with study requirements, as evaluated by history, physical exam, and chemistry panel. Participants with a prior (within 2 years of study therapy initiation) or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen will be permitted to join the study.