All-Trans Retinoic Acid (ATRA) Plus PD-1 Inhibition in Recurrent IDH-Mutant Glioma

Inclusion Criteria: 1. Prior histopathologically proven diagnosis of astrocytoma (grade 2-4) or oligodendroglioma (grade 2-3) according to the World Health Organization (WHO) 2021 Classification System that is progressive or recurrent following at least one prior alkylating chemotherapy regimen (i.e., temozolomide and/or lomustine), +/- radiation therapy 2. Patient's tumor must have a known mutation in IDH1 or IDH2. IDH1/2 mutation status must be confirmed by DNA sequencing and could have been performed in any CLIA/CAP-certified laboratory. IDH1/2 mutational testing could have been performed on patient's tumor either at initial diagnosis or on a subsequent recurrent tumor. 3. Safety Run-In and Phase 2 (Arm A and Arm B) patients: * All Safety Run-In and Phase 2 patients: patients with any contrast-enhancing tumor must have measurable disease per RANO criteria (defined by at least 1cmx1cm of contrast-enhancing tumor). Patients with exclusively non-enhancing tumors must have least a 25% increase in bi-dimensional product of FLAIR signal abnormality (measurable disease) compared to the patient's best MRI scan (smallest bi-dimensional product of FLAIR signal abnormality) obtained following completion of the patient's most recent line of therapy * Safety Run-In: Must have failed temozolomide OR another alkylator (e.g. lomustine, procarbazine, carmustine). May have failed an unlimited number of prior systemic regimens, +/- prior radiotherapy. * Arm A: Must have failed temozolomide AND another alkylator (e.g. lomustine, procarbazine, carmustine). May have failed an unlimited number of prior systemic regimens, +/- prior radiotherapy. * Arm B: Must have failed temozolomide OR another alkylator (maximum one prior chemotherapy regimen) +/- prior radiotherapy, AND must have gone at least 12 months since last treatment (chemotherapy or radiotherapy). 4. Surgical patients (Arm C and Arm D): * Must have clinical indication for surgical resection of the suspected recurrent/progressive tumor, as determined by patient's care providers; measurable disease is not required * 5-aminolevulinic acid (5-ALA) is not allowed for intraoperative tumor visualization due to the photosensitizing agent interaction with ATRA * Patient may have had an unlimited number of relapses and prior therapy regimens 5. Patients must be able to undergo MRI of the brain with gadolinium. Patients must be maintained on a stable or decreased dose of corticosteroid regimen (no increase for 5 days) prior to this baseline MRI. 6. Patients must have recovered from severe toxicity of prior therapy; the following intervals from previous treatments are required to be eligible: * 12 weeks from completion of radiation * 6 weeks from a nitrosourea cytotoxic chemotherapy * 3 weeks from a non-nitrosourea cytotoxic chemotherapy * 4 weeks from any investigational (not Food and Drug Administration \[FDA\]-approved) agents, or within a time interval less than at least 5 half-lives of the investigational agent, whichever is shorter * 2 weeks from administration of a non-cytotoxic, FDA-approved agent (e.g. abemaciclib, olaparib, etc) 7. If patient is on systemic corticosteroids to treat brain edema and/or brain edema-related symptoms, the dose must be 2mg of dexamethasone (or equivalent) daily or less for a minimum of 5 days prior to first dose of retifanlimab. 8. Patients must be able to swallow oral medications 9. Age 18 or older 10. Karnofsky performance status greater than or equal to 60 11. Life expectancy \>3 months 12. Adequate organ and marrow function: * Total bilirubin \<1.5 x upper limit of normal (ULN) (except patients with suspected Gilbert's Syndrome, who are eligible for the study but exempt from the total bilirubin eligibility criterion) * ALT and AST ≤ 2.5x ULN * Calculated CrCl ≥ 30 ml/min (glomerular filtration rate can also be used in place of CrCl) * Absolute Neutrophil count ≥1,500/uL * Platelets ≥ 100,000/uL * Hemoglobin ≥ 9 g/dL 13. Reproductive Status 1. Women of childbearing potential (WOCBP) must have a negative serum pregnancy test (minimum sensitivity 25 IU/L or equivalent units of HCG) within 7 days prior to the start of study drug. 2. b) Women must agree to not breastfeed during the study or for 180 days after the last dose of study treatment 3. WOCBP must agree to use an adequate method to avoid pregnancy (as defined below) from the time of study screening through 180 days from last dose of study drug 4. Males who are sexually active with WOCBP must agree to follow instructions for method(s) of contraception (as defined below) starting with the first dose of study drug through 180 days after the last dose of study 5. Azoospermic males and WOCBP who are continuously not heterosexually active are exempt from contraceptive requirements. However, these WOCBP must still undergo pregnancy testing as described in this section. At a minimum, participants of childbearing potential who are sexually active and their partners must agree to the use of a highly effective form of contraception (as defined below) throughout their participation beginning with the time of consent, during the study treatment, and for 180 days after last dose of study treatment(s). HIGHLY EFFECTIVE METHODS OF CONTRACEPTION: * Hormonal methods of contraception including combined oral contraceptive pills, vaginal ring, injectables, implants and intrauterine devices (IUDs) by WOCBP subject or male subject's WOCBP partner. Female partners of male subjects participating in the study may use hormone-based contraceptives as one of the acceptable methods of contraception since they will not be receiving study drug * Nonhormonal IUDs * Bilateral Tubal ligation * Vasectomy * Sexual Abstinence * It is not necessary to use any other method of contraception when complete abstinence is elected. * WOCBP participants who choose complete abstinence must continue to have pregnancy tests. * Acceptable alternate methods of highly effective contraception must be discussed in the event that the WOCBP participants chooses to forego complete abstinence. 14. Participant must, in the opinion of the Investigator, be able to comply with study procedures 15. Patients must be able to understand the study procedures and agree to participate in the study by providing written informed consent (or have legally authorized representative sign on patient's behalf if patient physically unable to sign consent due to neurologic deficit) Exclusion Criteria: Any of the following would exclude the subject from participation in the study: 1. Contrast-enhancing tumor in brainstem or spinal cord (subjects do not need spinal MRI for screening, but known spinal cord tumor is exclusionary) 2. Diffuse leptomeningeal disease 3. Patients who have received bevacizumab within the last 3 months are ineligible 4. Patients with clinically significant mass effect or midline shift (e.g., 1-2 cm of midline shift) 5. Use of any immunosuppressive medication other than steroids, including but not limited to antimetabolites, calcineurin inhibitors, and/or anti-TNF agents within six months of start of study drug 6. Prior diagnosis of immunodeficiency 7. Prior solid organ or bone marrow transplantation 8. Active autoimmune disease requiring systemic immunosuppression in excess of physiologic maintenance doses of corticosteroids (\> 10 mg/day of prednisone or equivalent). * Physiologic corticosteroid replacement therapy at doses ≤ 10 mg/day of prednisone or equivalent for adrenal or pituitary insufficiency and in the absence of active autoimmune disease is permitted. * Participants with asthma that requires intermittent use of bronchodilators, inhaled corticosteroids, or local corticosteroid injections may participate. * Participants using topical, ocular, intra-articular, or intranasal corticosteroids (with minimal systemic absorption) may participate. * Brief courses of corticosteroids for prophylaxis (eg, contrast dye allergy) or study treatment-related standard premedications are permitted. EXCEPTIONS: Patients with the following autoimmune diseases may participate: type I diabetes mellitus, hypothyroidism only requiring hormone replacement, Grave's disease that is previously treated with thyroidectomy or radioiodine, celiac disease with symptoms controlled with a gluten-free diet. 9. Evidence of interstitial lung disease, history of interstitial lung disease, or active, noninfectious pneumonitis. 10. Immune related toxicity during prior checkpoint inhibitor therapy for which permanent discontinuation of therapy was recommended (per product label or consensus guidelines) or any immune-related toxicity requiring intensive or prolonged immunosuppression to manage (with the exception of endocrinopathy that is well controlled on replacement hormones). 11. Known active hepatitis B virus (HBsAg reactive) or active hepatitis C virus (HCV RNA detectable by PCR) 12. Human immunodeficiency virus (HIV)-positive patients on antiretroviral therapy 13. Patients with a prior or concurrent malignancy whose natural history or treatment has the potential to interfere with the safety or efficacy assessment of the investigational regimen are excluded from this trial. Otherwise, patients with prior or concurrent malignancy are eligible. 14. Any serious, uncontrolled medical disorder, nonmalignant systemic disease, or active, uncontrolled infection that, in the opinion of the investigator, would put the subject at undue risk from the study treatment. 15. Patients with uncontrolled or significant cardiovascular disease including, but not limited to, any of the following are ineligible: * Myocardial infarction or uncontrolled angina within 90 days prior to consent * History of clinically significant arrhythmia (such as ventricular tachycardia, ventricular fibrillation, or torsades de pointes) * History of cardiomyopathy, pericarditis, significant pericardial effusion, myocarditis, or New York Heart Association (NYHA) functional class III-IV congestive heart failure 16. Known hypersensitivity to another monoclonal antibody that cannot be controlled with standard measures (e.g., antihistamines and corticosteroids) 17. Known allergy or hypersensitivity to any component of retifanlimab or formulation components. 18. Known allergy or hypersensitivity to all-trans retinoic acid (tretinoin), any of its components, or other retinoids 19. Prisoners or subjects who are involuntarily incarcerated 20. Subjects who are compulsorily detained for treatment of either a psychiatric or physical (eg, infectious disease) illness 21. Pregnant women are excluded 22. Has received a live vaccine within 28 days before the planned start of study treatment Note: Examples of live vaccines include but are not limited to measles, mumps, rubella, varicella-zoster (chickenpox), yellow fever, rabies, BCG, and typhoid vaccines. Seasonal influenza vaccines for injection are generally killed-virus vaccines and are allowed; however, intranasal influenza vaccines are live, attenuated vaccines and are not allowed. 23. Participant must not be simultaneously enrolled in any interventional clinical trial