A Trial Evaluating the Efficacy of Ivonescimab in Metastatic Endocrine Refractory HR-positive HER2-negative or Triple Negative Invasive Lobular Carcinoma

Inclusion Criteria: * 18 years of age or older. * Historically confirmed invasive lobular cancer with negative E-cadherin staining by IHC. * Estrogen receptor (ER) positive (\>1%) or negative, progesterone receptor (PgR) positive or negative, and HER2-negative according to HER2 testing guidelines from the American Society of Clinical Oncology/College of American Pathologists. * Participants must be willing to undergo biopsy as required by the study if the tumor is safely accessible. * If ER+, participant must be endocrine refractory as per the treating oncologist assessment and has been exposed to at least one line of endocrine therapy prior to enrollment. * Participants who received prior chemotherapy, antibody-drug conjugates (ADCs), mTOR inhibitor and/or PI3K are eligible. o Participants should not have received more than 2 chemotherapeutic agents and/or ADCs in the metastatic setting. * Eastern Cooperative Oncology Group Performance status ≤ 1. * Participant has either measurable disease per response evaluation criteria in solid tumors (RECIST) v1.1 criteria OR at least one predominantly lytic bone lesion must be present. * Adequate Organ Function: * Hematology (no blood transfusions or growth factor therapy used within 7 days of the screening CBC): i. Absolute neutrophil count (ANC) ≥ 1.0 × 109/L ii. Platelet count ≥ 100 × 109/L iii. Hemoglobin ≥ 9.0 g/dL * Kidneys: * Creatinine clearance (CrCL) ≥ 60 mL/min using the Cockcroft-Gault formula or estimated glomerular filtration rate (eGFR) value ≥60 mL/min using the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equation (adjustment by BSA is not required for eGFR). CrCL or eGFR can be determined using the calculator from the National Kidney Foundation website (www.kidney.org). * Urine protein \< 2+ or 24 hour urine protein quantification \< 1.0 g * Liver: i. Serum total bilirubin (TBIL) ≤ 1.5 × upper limit of normal (ULN); For patients with liver metastases or confirmed/suspected Gilbert syndrome, TBIL ≤3 × ULN ii. Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) ≤ 2.5 × ULN; For participants with liver metastases, AST and ALT ≤ 5 × ULN * Coagulation: prothrombin time (PT) or international normalized ratio (INR) ≤ 1.5 × ULN, and partial prothrombin time (PTT) or activated partial thromboplastin time (aPTT) ≤ 1.5 × ULN (unless abnormalities are unrelated to coagulopathy or prophylactic coagulation). This applies only to patients who are not on therapeutic anti-coagulation. Participants receiving therapeutic anti-coagulation should be on a stable dose. * Participants with active hepatitis B are required to have stable or declining levels of hepatitis B DNA by polymerase chain reaction (PCR) on appropriate anti-viral therapy with acceptable tolerability for one month prior to enrolment on study. * All participants with active hepatitis C (hepatitis C virus \[HCV\] antibody positive with HCV RNA levels above the lower limit of detection) are excluded. * Participants with treated brain metastases are eligible if follow-up brain imaging after central nervous system (CNS)-directed therapy shows no evidence of progression and must have stopped corticosteroids or be on physiologic corticosteroid replacement therapy (prednisone ≤ 10 mg daily or equivalent) * Participants with a prior or concurrent malignancy whose natural history or treatment does not interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial. * The effects of ivonescimab on the developing human fetus are unknown. For this reason, women of child-bearing potential (WOCBP) and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation. (Refer to Pregnancy Assessment Policy MD Anderson Institutional Policy # CLN1114). This includes all female patients, between the onset of menses (as early as 8 years of age) and 55 years unless the participant presents with an applicable exclusionary factor which may be one of the following: * Postmenopausal (no menses in greater than or equal to 12 consecutive months). * History of hysterectomy or bilateral salpingo-oophorectomy. * Ovarian failure (Follicle Stimulating Hormone and Estradiol in menopausal range, who have received Whole Pelvic Radiation Therapy). * History of bilateral tubal ligation or another surgical sterilization procedure. * Approved methods of birth control are as follows: Hormonal contraception (i.e. birth control pills, injection, implant, transdermal patch, vaginal ring), Intrauterine device (IUD), Tubal Ligation or hysterectomy, Participant/Partner post vasectomy, Implantable or injectable. contraceptives, and condoms plus spermicide. Not engaging in sexual activity for the total duration of the trial and the drug washout period is an acceptable practice; however periodic abstinence, the rhythm method, and the withdrawal method are not acceptable methods of birth control. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. * WOCBP must have a negative urine pregnancy test on the day of first dose prior to dosing. * WOCBP must agree to use adequate contraception for the duration of study treatment and 30 days after the last dose of study treatment. * Unsterilized male participants having sex with a female partner of childbearing potential, or a pregnant or breastfeeding partner must agree to use barrier contraception (male condom) for the duration of the treatment period until 30 days after the last dose of ivonescimab. Male patients with female partners of childbearing potential must have the female partner agree to use at least 1 form of highly effective contraception for the duration of the treatment period until 30 days after the last dose of ivonescimab * Ability to understand and the willingness to sign a written informed consent document. Exclusion Criteria: Participants who have had chemotherapy or radiotherapy within 4 weeks prior to entering the study. * Participants who are receiving any other investigational agents. * Major surgical procedures or serious trauma within 4 weeks prior to treatment start date or plans for major surgical procedures within 4 weeks after the first dose (as determined by the investigator). Minor local procedures (excluding central venous catheterization and port implantation) within 3 days prior to enrolment. * Participants with symptomatic CNS metastases, CNS metastases with hemorrhagic features, CNS metastasis ≥ 1.5 cm, CNS radiation within 7 days prior to enrolment potential need for CNS radiation within the first cycle, or leptomeningeal disease. * History of bleeding tendencies or coagulopathy and/or clinically significant bleeding symptoms or risk within 4 weeks prior to enrolment on study, including but not limited to: * Clinically significant bleeding (e.g., coughing up ≥0.5 teaspoon of fresh blood or small blood clots). Brief or transient bleeding associated with diagnostic procedures (e.g., endoscopy or bronchoscopy) is permitted. * Nasal bleeding /epistaxis (bloody nasal discharge is allowed) * Current use of prophylactic or full-dose anticoagulants or anti-platelet agents for therapeutic purposes that is not stable prior to enrolment on study is not allowed. The use of full-dose anticoagulants is permitted as long as the international normalized ratio (INR) or activated partial thromboplastin time (aPTT) is within therapeutic limits according to the medical standard of the enrolling institution. * Poorly controlled hypertension with repeated systolic blood pressure ≥ 150 mmHg or diastolic blood pressure ≥ 100 mmHg after oral antihypertensive therapy * Live vaccine or live attenuated vaccine within 4 weeks prior to planned enrolment on study, or if scheduled to receive a live vaccine or live attenuated vaccine during the study period. Inactivated vaccines are permitted. * Severe infection within 4 weeks prior to enrolment on study, including but not limited to comorbidities requiring hospitalization, sepsis, or severe pneumonia; active infection (as determined by the investigator) requiring systemic anti-infective therapy within 2 weeks prior to enrolment on study (excluding antiviral therapy for hepatitis B or C) * Has pre-existing peripheral neuropathy that is ≥ Grade 2 by CTCAE version 5 * Uncontrolled pleural effusions, pericardial effusions, or ascites that is clinically symptomatic. Participants managed with indwelling catheters (eg, PleurX) are allowed. * History of non-infectious pneumonia requiring systemic corticosteroids, or current interstitial lung disease * Active or prior history of inflammatory bowel disease (eg, Crohn's disease, ulcerative colitis, or chronic diarrhea) * Active autoimmune or lung disease requiring systemic therapy (eg, with disease modifying drugs, prednisone \>10 mg daily or equivalent, immunosuppressant therapy) within 2 years prior to enrolment, however the following will be allowed: * Replacement therapy (eg, thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency) is permitted. * Intermittent use of bronchodilators, inhaled corticosteroids, or local corticosteroid injections is permitted * History of major diseases before enrolment on study, specifically: * Unstable angina, myocardial infarction, congestive heart failure (New York Heart Association \[NYHA\] classification ≥ grade 2) or unstable vascular disease (eg, aortic aneurysm at risk of rupture, Moyamoya disease) that required hospitalization within 12 months prior tto enrolment on study, or other cardiac impairment that may affect the safety evaluation of the study drug (eg, poorly controlled arrhythmias, myocardial ischemia) * History of esophageal gastric varices, severe ulcers, wounds that do not heal, abdominal fistula, intra-abdominal abscesses, or acute gastrointestinal bleeding within 6 months before enrolment * History of any grade arterial thromboembolic event, venous thromboembolic event of Grade 3 and above as specified in National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) 5.0, transient ischemic attack, cerebrovascular accident, hypertensive crisis, or hypertensive encephalopathy within 12 months prior to enrolment. * Acute exacerbation of chronic obstructive pulmonary disease within 4 weeks before enrolment. * History of perforation of the gastrointestinal tract and/or fistula, history of gastrointestinal obstruction (including incomplete intestinal obstruction requiring parenteral nutrition), extensive bowel resection (partial colectomy or extensive small bowel resection) within 6 months prior to enrolment. * Imaging during the screening period shows that the patient has: * Radiologically documented evidence of major blood vessel invasion or tumor invading critical organs or structures (e.g., heart, trachea, esophagus), or a risk of fistula formation (such as esophagotracheal or esophagopleural) in the opinion of the investigator. * Radiographic evidence of significant blood vessel encasement with narrowing of the vessel or evidence of intratumoral cavitation or necrosis that, in the judgment of the investigator, would pose a substantially increased risk of bleeding. * Human immunodeficiency virus (HIV)-infected patients on effective anti-retroviral therapy with undetectable viral load within 6 months are eligible for this trial. * Participants with psychiatric illness/social situations that would limit compliance with study requirements. * Pregnant women are excluded from this study because ivonescimab is an anti-cancer agent with the potential for teratogenic or abortifacient effects. * Participant is breastfeeding or plans to breastfeed during the study * Known allergy to any component of any study drug * Known history of severe hypersensitivity to other monoclonal antibodies * History or current evidence of any condition (medical \[including adverse events from prior anticancer therapy, disorders secondary to tumor\], surgical or psychiatric \[including substance abuse\]), or laboratory abnormality that might confound the results of the study, interfere with the participant's participation for the full duration of the study, might lead to higher medical risk and/or is not in the best interest of the participant to participate, in the opinion of the treating investigator.