Pembrolizumab plus Radiation for Bladder Cancer

Inclusion Criteria: * Pathologically (histologically) proven diagnosis of T1 high-grade non-muscle invasive urothelial carcinoma of the bladder without radiographic evidence of regional nodal disease or metastatic disease (N0, M0) on CT, MRI, or positron emission tomography (PET)/CT scan who would otherwise be treated with cystectomy off-trial. Patients should have cystectomy recommended disease but do not need to be medically operable for a cystectomy to be eligible for the trial. * NOTE: Patients with nodal disease ≥ 1 cm on short-axis or with suspicious nodes that are PET-avid of any size are not eligible * High grade T1 disease history that must meet at least ONE of the three criteria below: * Histologically confirmed recurrence with high-grade T1 urothelial carcinoma (+/- focal carcinoma in situ \[CIS\]) in the bladder following initial transurethral resection of bladder tumor (TURBT) and at least one induction course of intravesical therapy. Adequate induction course is defined as ≥ 5 doses of intravesical Bacillus Calmette-Guerin (BCG) or intravesical chemotherapy when BCG is not available. * T1 with pathologic high-risk features (lymphovascular invasion \[LVI\] or variant histology of micropapillary, sarcomatoid, or plasmacytoid features) post initial TURBT. (No prior intravesical therapy required) * Persistent high-grade T1 urothelial carcinoma at repeat TURBT (+/- focal CIS) in the bladder. (No prior intravesical therapy required) * Restaging TURBT must be performed and must meet ALL of the following criteria below: * If there is absence of muscularis propria in the initial TURBT, there must be uninvolved muscularis propria in the restaging TURBT. * All grossly visible papillary tumors must be removed * Note: If the restaging TURBT is performed outside of the enrolling institution, an office cystoscopy should be performed by a Urologist who will be following the patient as part of the clinical trial * No pure squamous cell carcinoma or adenocarcinoma of the bladder * No neuroendocrine (small or large cell) features * No diffuse carcinoma in situ determined on cystoscopy and biopsy (i.e. extensive carcinoma in situ that is not just tumor-associated CIS in the opinion of the site investigator) * No prostatic urethral involvement * Age ≥ 18 * Eastern Cooperative Oncology Group (ECOG) performance status of 0-2 * Negative urine or serum pregnancy test (in persons of childbearing potential) within 14 days prior to registration. Childbearing potential is defined as any person who has experienced menarche and who has not undergone surgical sterilization (hysterectomy or bilateral oophorectomy), tubal ligation or who is not postmenopausal * Absolute neutrophil count (ANC) ≥ 1,500 cells/mm\^3 * Platelets ≥ 100,000 cells/mm\^3 * Hemoglobin ≥ 9 g/dl (Note: The use of transfusion or other intervention to achieve hemoglobin \[Hgb\] ≥ 9 g/dl is acceptable) * Adequate renal function defined as creatinine clearance (CrCL) of ≥ 30 mL/min by the Cockcroft-Gault formula, ≤ 1.5 × upper limit of normal (ULN) or creatinine levels \> 1.5 × institutional ULN * Total bilirubin ≤ institutional upper limit of normal (ULN) (Not applicable to patients with known Gilbert's syndrome) * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\]) and alanine aminotransferase (ALT) (serum glutamic pyruvic transaminase \[SGPT\]) ≤ 3 x institutional ULN * All adverse events of their most recent therapy/intervention must have resolved to \< grade 3 or returned to baseline prior to registration * No history of pelvic radiation therapy * No prior systemic chemotherapy or immunotherapy for urothelial carcinoma. Prior treatment with local intravesical therapy including BCG or chemotherapy is allowed * No prior treatment with anti-PD-1, anti PD-L1, anti PD-L2 or anti-CTLA4 antibody or any other antibody or drug targeting T-cell co-stimulation * No live vaccine administered within 30 days of registration. All non live vaccines (including the coronavirus disease \[COVID\] vaccine) are allowed at any time during the study. Timing should minimize confusion with drug-related toxicities where possible * Patients must have recovered from acute cardiac illness * New York Heart Association Functional Classification II or better (New York Heart Association \[NYHA\] Functional Classification III/IV are not eligible) (Note: Patients with known history or current symptoms of cardiac disease, or history of treatment with cardiotoxic agents, should have a clinical risk assessment of cardiac function using the New York Heart Association Functional Classification.) * No active infection requiring IV antibiotics * No active autoimmune disease that required systemic treatment in the past 2 years (i.e., with use of disease modifying agents, corticosteroids, or immunosuppressive drugs). Replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment * No history of idiopathic pulmonary fibrosis, organizing pneumonia, (non-infectious) pneumonitis that required steroids or current pneumonitis * No history of allogeneic bone marrow transplant or prior solid organ transplant * No active tuberculosis * No evidence of hydronephrosis * No history of upper tract urothelial carcinoma within 24 months of registration * No patients with a prior diagnosis of prostate cancer who have not received definitive treatment for their prostate cancer (e.g. on active surveillance) * Patients with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial * No glucocorticoids except physiologic doses are allowed. The use of doses of corticosteroids (defined as 10 mg prednisone or equivalent) is acceptable * No history of allergic reaction to the drug excipients