Skip to content

fix(xlsx): keep currency labels from number formats - #2538

Open
Islam El-Nashar (aslamalkarywk7) wants to merge 4 commits into
microsoft:mainfrom
aslamalkarywk7:fix-xlsx-currency-labels
Open

Islam El-Nashar (aslamalkarywk7) wants to merge 4 commits into
microsoft:mainfrom
aslamalkarywk7:fix-xlsx-currency-labels

Conversation

@aslamalkarywk7

Copy link
Copy Markdown

Closes #53

pandas.read_excel returns raw values and drops Excel number formats, so currency-formatted cells lost their label (1199 instead of ). XlsxConverter now overlays the currency symbol from each cell's number_format (quoted literals and locale blocks like [], prefix or suffix by format order) onto the values before rendering, without touching the numbers themselves.

Verification: new tests/test_xlsx_currency.py (3 tests: $ prefix, untouched plain cells, EUR suffix) pass; existing xlsx image tests unaffected (22 pass; 1 pre-existing env failure for missing xlrd also fails on main); full module-vectors failure count identical before/after (41 pre-existing env failures from missing optional deps).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both helpers always inspect the first semicolon section, then apply that currency to every numeric cell. Negative values use the second Excel format section (zero can use the third), so a format like "$"#,##0;"€"#,##0 will render a negative value with $. The selected section needs to depend on the cell value, or section-specific currencies should be left alone.

@aslamalkarywk7

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Negative values use the second Excel format section (zero can use
the third), so inspect the section matching the cell sign instead
of always using the first section.
@aslamalkarywk7

Copy link
Copy Markdown
Author

Thanks for pointing this out. I’ve fixed the issue by selecting the Excel number-format section based on the cell value:

  • Positive values use the first section.
  • Negative values use the second section.
  • Zero values use the third section when present.

This ensures that section-specific currencies are preserved correctly, for example:

"$"#,##0;"€"#,##0

will render positive values with $ and negative values with . I also added tests covering these cases.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked da00881. Currency extraction and prefix placement now use the number-format section selected from the cell value, so positive, negative and zero values no longer all inherit the first section's currency. The regression covers the exact $/€ split I raised plus a third zero section. My concern is resolved.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Currency detection has parsing gaps, one assertion is ineffective, and Black formatting will fail CI.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 4 Medium severity

Open (4)
What changed in this PR

Adds currency labels from XLSX number formats to converted Markdown.

Changes:

  • Detects currency symbols and their placement.
  • Overlays labels onto pandas cell values.
  • Adds currency conversion tests.
File Description
_xlsx_converter.py Adds currency-format parsing and label overlay.
test_xlsx_currency.py Tests prefix, suffix, plain, and sectioned formats.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

# Currency symbols that may appear in Excel number formats, either as quoted
# literals (e.g. '"$"#,##0.00') or locale blocks (e.g. '[$€-x-euro2]').
# See https://github.com/microsoft/markitdown/issues/53.
_CURRENCY_SYMBOL_RE = re.compile(r'[$€£¥₹₽₩₪₺₴₸₫₦¤]')
Excel uses: 1 section = all numbers, 2 sections = positive+zero / negative,
3 sections = positive / negative / zero (4th section is text and ignored).
"""
parts = number_format.split(";")
def _overlay_currency_labels(
sheets: dict[str, Any], workbook_stream: BinaryIO
) -> None:
"""Rewrite currency-formatted numeric cells with their display label.
markdown = _convert(stream.getvalue())
assert "$10" in markdown
assert "€" in markdown
assert "$-5" not in markdown.replace("$-5", "") or "€" in markdown
@afourney

Copy link
Copy Markdown
Member

Thanks for working on this. It seems there is some additional complexity to this problem that I need to think about.

I had Astra review this PR and it suggested the following, which I am now confirming by hand:

f4a7ed1 restores the dollar labels in both workbooks attached to #53. The updated positive/negative/zero section handling also addresses the earlier review concern.

However, there are several cases where the implementation introduces incorrect currency information. I’m requesting changes before merging.

  1. Locale metadata is mistaken for a displayed dollar sign.

    In _currency_symbol, the fallback searches the entire format string for currency symbols. The $ in a locale-only block such as [$-409] is formatting syntax, not a displayed currency symbol.

    I reproduced these results through XlsxConverter:

    Value Number format Current main PR output
    42 [$-409]#,##0.00 42 $42
    0.42 [$-409]0% 0.42 $0.42

    The second example incorrectly labels a percentage as money. Please distinguish the currency payload of a locale block from the syntax introducing that block. An empty currency payload must not produce $.

  2. Multi-character currency labels are truncated.

    _currency_symbol returns a single matching character, discarding parts of the label that identify the currency:

    Number format, with value 42 PR output Lost information
    "R$" #,##0.00 $42 Brazilian-real qualifier
    [$A$-en-AU]#,##0.00 $42 Australian-dollar qualifier
    [$USD-409]#,##0.00 42 Entire currency code
    #,##0.00 "CHF" 42 Entire currency code

    Please preserve the complete recognized currency label. Detecting Unicode currency symbols would address some omissions in the current allowlist, but would not resolve these cases.

  3. Conditional formats select the wrong section.

    _select_format_section assumes that sections are selected exclusively by the value’s sign. Explicit conditions override those rules.

    For example, with the format [>=100]"$"0;"€"0, a value of 50 should use the euro section. The PR produces 50$.

    Similarly, [>=100]"$"0;0 incorrectly labels 50 as currency even though the applicable section has no currency label.

    Please handle explicit conditions correctly, or conservatively leave conditional formats unchanged until supported.

  4. Prefix/suffix detection treats metadata as displayed content.

    _is_currency_position_prefix searches the raw format for the first currency character and numeric placeholder. Characters inside bracketed metadata interfere with both searches:

    Value Number format PR output Intended placement
    150 [Color10]"$"#,##0 150$ Prefix
    150 [>=100]"$"0;"€"0 150$ Prefix
    42 [$-409]#,##0.00"€" €42 Suffix

    In the first two cases, the 0 inside the metadata is mistaken for a numeric placeholder. In the third, the locale marker’s $ is mistaken for the position of the currency.

    Please determine placement using the actual currency token and numeric placeholders, excluding metadata and quoted or escaped literal contents.

There are also several smaller correctness and test-coverage gaps:

  • Splitting directly on ; breaks quoted or escaped semicolons. For example, the single-section format "$;gross"#,##0 loses its currency label for negative values.
  • The allowlist omits symbols such as ฿ and .
  • $0" net" loses its dollar label because the unrelated quoted literal prevents the fallback search.
  • The second workbook read trusts declared worksheet dimensions, whereas pandas resets them. With actual data through row 3 but a stale dimension of A1:A2, only the first data row receives currency labels. Please make the overlay’s iteration consistent with the data pandas actually read.
  • The final assertion in test_section_specific_currency_uses_cell_value is always true: replace("$-5", "") removes every occurrence before checking for its absence. Please check the original Markdown and assert the expected negative-value output directly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unable to extract currency from excel formatted cells

4 participants