diff --git a/CHANGELOG.md b/CHANGELOG.md index 760da1d1..ee3eb498 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -40,6 +40,13 @@ ### Fixes +- `for` and `do` loops now reject misspelled `in` and `while` keywords before any statements in the input execute. They previously accepted any identifier in those positions. +- `filter` now rejects array index literals larger than `2147483647`, including optional access, instead of silently selecting element zero. Representable indexes beyond the array length still return `null`. +- Decimal arithmetic and comparisons involving JSON numeric strings now use the same culture-invariant `double` conversion as shell strings instead of failing with a `Decimal`-to-`Double` cast error. String concatenation with `+` is unchanged. +- `exec` now accepts normal command options, negative arguments, URLs, and file patterns while preserving variable, JSON, and parenthesized-expression arguments. Options previously failed to parse, including the documented `exec $cmd -m 5` form. +- `mkdb`, `mkcon`, `create database`, and `create container` now work on serverless accounts. They previously requested autoscale throughput even when `--scale` and `--ru` were omitted, which serverless accounts reject. Omitting both options now creates the resource without throughput settings; supplying either option on a serverless account fails with an explanation. Provisioned accounts keep the existing autoscale default of 1000 RU/s. ([#218](https://github.com/Azure/CosmosDBShell/issues/218)) +- Vector `ORDER BY`, `ORDER BY RANK` relevance ranking, and object-shaped `DISTINCT` projections no longer fail with a continuation-token error. These query pipelines execute successfully but cannot export a resumable token, which was previously reported as a command failure. Such queries now return their documents; through MCP they keep reading until the requested limit instead of stopping after one page, and a truncated result is reported as `resultIncomplete` rather than as an exhausted result set. ([#219](https://github.com/Azure/CosmosDBShell/issues/219)) +- Local emulator outages are now detected across Cosmos DB commands. Requests fail promptly with an error and return the shell to its disconnected state instead of leaving an unresponsive session labeled as connected. - Explicit JSON `null` values in ordinary MCP arguments are treated as omitted rather than failing nullable option binding, and are not echoed into history. Required arguments still fail validation when absent. `continuation` and the `rm` safety options `partition-key`/`pk` and `etag` reject explicit nulls instead of silently removing paging or deletion safeguards. Numeric arguments in echoed commands, confirmations, and history now use invariant formatting so they replay correctly under comma-decimal locales. ([#227](https://github.com/Azure/CosmosDBShell/pull/227)) - Concurrent shell processes now merge saved history under a shared lock instead of overwriting each other's entries. Saves and clears publish a complete, flushed replacement, preserving the existing file on failed writes; failed clears report an error without discarding loaded history. Temporary history files use restricted permissions, including preserved access rules or owner-only access on Windows. ([#228](https://github.com/Azure/CosmosDBShell/pull/228)) - CSV export now preserves scalar, array, and null query results in a scalar column, including results mixed with objects. The header is empty by default and can be named with `COSMOSDB_SHELL_CSV_SCALAR_COLUMN`; a collision with an object property fails explicitly. CSV import ignores empty headers only for empty cells and reports a line/column error for non-empty values under an empty header. ([#224](https://github.com/Azure/CosmosDBShell/pull/224)) diff --git a/CosmosDBShell.Tests/CommandTests/FilterCommandTests.cs b/CosmosDBShell.Tests/CommandTests/FilterCommandTests.cs index 86cfd865..143d11a0 100644 --- a/CosmosDBShell.Tests/CommandTests/FilterCommandTests.cs +++ b/CosmosDBShell.Tests/CommandTests/FilterCommandTests.cs @@ -11,6 +11,23 @@ namespace CosmosShell.Tests.CommandTests; public class FilterCommandTests { + [Theory] + [InlineData(".[2147483648]")] + [InlineData(".[2147483648]?")] + public async Task ExecuteAsync_OverflowingIndex_RejectsInputWithoutChangingResult(string expression) + { + using var shell = ShellInterpreter.CreateInstance(); + var input = new ShellJson(JsonSerializer.SerializeToElement(new[] { 1, 2, 3 })); + var state = new CommandState { Result = input }; + var command = new FilterCommand { ExpressionText = expression }; + + var error = await Assert.ThrowsAsync(() => + command.ExecuteAsync(shell, state, string.Empty, TestContext.Current.CancellationToken)); + + Assert.Contains("2147483648", error.Message); + Assert.Same(input, state.Result); + } + [Fact] public async Task ExecuteAsync_AppliesPathExpression_AndPreservesStructuredResult() { diff --git a/CosmosDBShell.Tests/Parser/CommandExecutionTests.cs b/CosmosDBShell.Tests/Parser/CommandExecutionTests.cs index 3c6a87f8..6bb63f94 100644 --- a/CosmosDBShell.Tests/Parser/CommandExecutionTests.cs +++ b/CosmosDBShell.Tests/Parser/CommandExecutionTests.cs @@ -92,6 +92,74 @@ public async Task Exec_VariableCommand_RunsResolvedCommand() Assert.Equal("hello world", Assert.IsType(state.Result).Text); } + [Theory] + [InlineData("-5", "-5")] + [InlineData("-1.5 +2", "-1.5 +2")] + [InlineData("\"--format=json\"", "--format=json")] + [InlineData("https://localhost:8081 *.json", "https://localhost:8081 *.json")] + public async Task Exec_ShellWords_RunLikeDirectArguments(string arguments, string expected) + { + var state = await Shell.RunCommandAsync(new(), $"exec \"echo\" {arguments}", TestContext.Current.CancellationToken); + + Assert.False(state.IsError); + Assert.Equal(expected, Assert.IsType(state.Result).Text); + } + + [Theory] + [InlineData("-directory . -l")] + [InlineData("--directory=. --list")] + [InlineData("--directory:. --list=false")] + [InlineData("--directory=$path --list=true")] + public async Task Exec_Options_BindAndRunResolvedCommand(string arguments) + { + SetVariable("cmd", new ShellText("dir")); + SetVariable("path", new ShellText(".")); + + var state = await Shell.RunCommandAsync( + new(), $"exec $cmd \"*.missing-regression-file\" {arguments}", TestContext.Current.CancellationToken); + + Assert.False(state.IsError); + Assert.Equal(0, Assert.IsType(state.Result).Value.GetProperty("values").GetArrayLength()); + } + + [Fact] + public async Task Exec_FunctionArguments_PreserveTypesAndParenthesizedExpressions() + { + var state = await Shell.RunCommandAsync( + new(), + "def capture [object flag number] { return [$object,$flag,$number] }; " + + "exec \"capture\" {\"id\":1} true (1.5 + 2)", + TestContext.Current.CancellationToken); + + Assert.False(state.IsError); + Assert.Equal("[{\"id\":1},true,3.5]", Assert.IsType(state.Result).Value.GetRawText()); + } + + [Fact] + public async Task Exec_UnknownOption_Throws() + { + await Assert.ThrowsAsync(() => RunSingleAsync("exec \"dir\" --bogus=1")); + } + + [Fact] + public async Task Exec_OptionMissingValue_Throws() + { + await Assert.ThrowsAsync(() => RunSingleAsync("exec \"dir\" --directory")); + } + + [Theory] + [InlineData("exec \"echo\" first; echo after")] + [InlineData("exec \"echo\" first\necho after")] + [InlineData("{ exec \"echo\" first }; echo after")] + [InlineData("exec \"echo\" first | echo after")] + public async Task Exec_ArgumentTerminators_PreserveFollowingStatements(string source) + { + var state = await Shell.RunCommandAsync(new(), source, TestContext.Current.CancellationToken); + + Assert.False(state.IsError); + Assert.Equal("after", Assert.IsType(state.Result).Text); + } + [Fact] public async Task Exec_EmptyCommandPath_Throws() { diff --git a/CosmosDBShell.Tests/Parser/CultureInvariantConversionTests.cs b/CosmosDBShell.Tests/Parser/CultureInvariantConversionTests.cs index 1e7c05d8..decddd68 100644 --- a/CosmosDBShell.Tests/Parser/CultureInvariantConversionTests.cs +++ b/CosmosDBShell.Tests/Parser/CultureInvariantConversionTests.cs @@ -5,6 +5,7 @@ namespace CosmosShell.Tests.Parser; using System.Globalization; +using System.Text.Json; using System.Threading; using System.Threading.Tasks; @@ -51,6 +52,30 @@ public async Task ExpressionEvaluation_TextDecimalArithmetic_UsesInvariantCultur Assert.Equal(3.0d, value.Value); } + [Theory] + [InlineData("1.5", 1.5)] + [InlineData("\"1.5\"", 1.5)] + [InlineData("1e100", 1e100)] + [InlineData("\"1e100\"", 1e100)] + public void ShellJson_DecimalConversion_ReturnsInvariantDouble(string source, double expected) + { + using var document = JsonDocument.Parse(source); + var value = WithGermanCulture(() => + Assert.IsType(new ShellJson(document.RootElement).ConvertShellObject(DataType.Decimal))); + + Assert.Equal(expected, value); + } + + [Fact] + public void ShellJson_DecimalConversion_RejectsLocalizedDecimalSeparator() + { + using var document = JsonDocument.Parse("\"1,5\""); + + WithGermanCulture(() => + Assert.Throws(() => + new ShellJson(document.RootElement).ConvertShellObject(DataType.Decimal))); + } + private static T WithGermanCulture(Func action) { var originalCulture = CultureInfo.CurrentCulture; diff --git a/CosmosDBShell.Tests/Parser/FilterPathExpressionTests.cs b/CosmosDBShell.Tests/Parser/FilterPathExpressionTests.cs index d072424a..e8ae883c 100644 --- a/CosmosDBShell.Tests/Parser/FilterPathExpressionTests.cs +++ b/CosmosDBShell.Tests/Parser/FilterPathExpressionTests.cs @@ -70,6 +70,35 @@ public async Task Index_OutOfRange_ReturnsNull() Assert.Equal(JsonValueKind.Null, Assert.IsType(result).Value.ValueKind); } + [Theory] + [InlineData(".[2147483647]")] + [InlineData(".[2147483647]?")] + public async Task Index_MaximumRepresentableIndex_ReturnsNull(string expression) + { + var result = await EvalAsync(expression, new[] { 10, 20, 30 }); + + Assert.Equal(JsonValueKind.Null, Assert.IsType(result).Value.ValueKind); + } + + [Theory] + [InlineData(".[2147483648]", "2147483648")] + [InlineData(".[2147483648]?", "2147483648")] + [InlineData(".items[999999999999999999999999]", "999999999999999999999999")] + public void Index_Overflow_ReportsErrorAtIndex(string expression, string index) + { + var lexer = new Lexer(expression); + var parser = new ExpressionParser(lexer); + + parser.ParseFilterExpression(); + + var error = Assert.Single(lexer.Errors); + Assert.Equal(expression.IndexOf(index, StringComparison.Ordinal), error.Start); + Assert.Equal(index.Length, error.Length); + Assert.Equal( + Azure.Data.Cosmos.Shell.Util.MessageService.GetArgsString("expression_error_invalid_number", "value", index), + error.Message); + } + [Fact] public async Task Index_OnNonArrayWithoutOptional_Throws() => await Assert.ThrowsAsync(() => EvalAsync(".[0]", new { a = 1 })); diff --git a/CosmosDBShell.Tests/Parser/ShellObjectConversionTests.cs b/CosmosDBShell.Tests/Parser/ShellObjectConversionTests.cs index 84d16489..3aa7e918 100644 --- a/CosmosDBShell.Tests/Parser/ShellObjectConversionTests.cs +++ b/CosmosDBShell.Tests/Parser/ShellObjectConversionTests.cs @@ -116,7 +116,7 @@ public void ShellJson_NumberToAllTypes() var json = Json("42"); Assert.Equal("42", json.ConvertShellObject(DataType.Text)); Assert.Equal(42, json.ConvertShellObject(DataType.Number)); - Assert.Equal(42m, json.ConvertShellObject(DataType.Decimal)); + Assert.Equal(42d, Assert.IsType(json.ConvertShellObject(DataType.Decimal))); Assert.Equal(true, json.ConvertShellObject(DataType.Boolean)); } @@ -170,7 +170,7 @@ public void ShellJson_StringToNumber(string raw, int expected) [Fact] public void ShellJson_StringToDecimal() { - Assert.Equal(15m, Json("\"15\"").ConvertShellObject(DataType.Decimal)); + Assert.Equal(15d, Assert.IsType(Json("\"15\"").ConvertShellObject(DataType.Decimal))); } [Fact] diff --git a/CosmosDBShell.Tests/Parser/StatementExecutionTests.cs b/CosmosDBShell.Tests/Parser/StatementExecutionTests.cs index 22ade0c4..d54f3309 100644 --- a/CosmosDBShell.Tests/Parser/StatementExecutionTests.cs +++ b/CosmosDBShell.Tests/Parser/StatementExecutionTests.cs @@ -30,6 +30,24 @@ public static System.Collections.Generic.IEnumerable ValueOriginCases( ("3.5", "+", "2", "Decimal", "5.5"), ("3", "/", "2", "Number", "1"), ("3.0", "/", "2", "Decimal", "1.5"), + ("3.0", "-", "\"2.5\"", "Decimal", "0.5"), + ("\"2.5\"", "-", "3.0", "Decimal", "-0.5"), + ("3.0", "*", "\"2.5\"", "Decimal", "7.5"), + ("\"2.5\"", "*", "3.0", "Decimal", "7.5"), + ("3.0", "/", "\"2.5\"", "Decimal", "1.2"), + ("\"2.5\"", "/", "3.0", "Decimal", "0.8333333333333334"), + ("3.0", "%", "\"2.5\"", "Decimal", "0.5"), + ("\"2.5\"", "%", "3.0", "Decimal", "2.5"), + ("3.0", "**", "\"2.5\"", "Decimal", "15.588457268119896"), + ("\"2.5\"", "**", "3.0", "Decimal", "15.625"), + ("3.0", "<", "\"2.5\"", "Boolean", "false"), + ("\"2.5\"", "<", "3.0", "Boolean", "true"), + ("3.0", "<=", "\"2.5\"", "Boolean", "false"), + ("\"2.5\"", "<=", "3.0", "Boolean", "true"), + ("3.0", ">", "\"2.5\"", "Boolean", "true"), + ("\"2.5\"", ">", "3.0", "Boolean", "false"), + ("3.0", ">=", "\"2.5\"", "Boolean", "true"), + ("\"2.5\"", ">=", "3.0", "Boolean", "false"), ("[1]", "+", "[2]", "Json", "[1,2]"), ("\"a\"", "==", "\"a\"", "Boolean", "true"), ("true", "&&", "false", "Boolean", "false"), @@ -96,6 +114,19 @@ public async Task SyntaxError_PreventsEarlierAssignment() Assert.Equal(1, GetInt("value")); } + [Theory] + [InlineData("for $x typo [1,2] { $value = 3 }")] + [InlineData("do { $value = 3 } typo false")] + public async Task InvalidLoopKeyword_PreventsEarlierAssignmentAndLoopBody(string invalid) + { + SetVariable("value", new ShellNumber(1)); + + var state = await Shell.RunCommandAsync(new(), $"$value = 2; {invalid}", TestContext.Current.CancellationToken); + + Assert.True(state.IsError); + Assert.Equal(1, GetInt("value")); + } + [Theory] [InlineData("break")] [InlineData("continue")] diff --git a/CosmosDBShell.Tests/Parser/StatementParserEdgeTests.cs b/CosmosDBShell.Tests/Parser/StatementParserEdgeTests.cs index 9736af15..aca8ffb5 100644 --- a/CosmosDBShell.Tests/Parser/StatementParserEdgeTests.cs +++ b/CosmosDBShell.Tests/Parser/StatementParserEdgeTests.cs @@ -49,6 +49,65 @@ public void Exec_WithArguments_ProducesExecStatement() Assert.IsType(Assert.Single(statements)); } + [Theory] + [InlineData("for $x typo [1,2] {}", "typo", "statement_error_expected_in")] + [InlineData("do {} typo false", "typo", "statement_error_expected_while")] + public void Loop_InvalidKeyword_ReportsErrorAtKeyword(string source, string keyword, string messageKey) + { + var (_, parser) = ParseWithParser(source); + + var error = Assert.Single(parser.Errors); + Assert.Equal(source.IndexOf(keyword, StringComparison.Ordinal), error.Start); + Assert.Equal(keyword.Length, error.Length); + Assert.Equal(Azure.Data.Cosmos.Shell.Util.MessageService.GetString(messageKey), error.Message); + } + + [Theory] + [InlineData("for $x IN [1,2] {}")] + [InlineData("do {} WHILE false")] + public void Loop_KeywordsRemainCaseInsensitive(string source) + { + var (statements, parser) = ParseWithParser(source); + + Assert.Single(statements); + Assert.Empty(parser.Errors); + } + + [Theory] + [InlineData("-directory . -r")] + [InlineData("--directory=. --recursive")] + [InlineData("--directory:. --recursive=false")] + [InlineData("--directory=$path")] + [InlineData("-5 -1.5 +2")] + [InlineData("https://localhost:8081 *.json \"two words\"")] + [InlineData("{\"id\":1} [1,2] true $value (1 + 2)")] + public void Exec_ArgumentsMatchDirectCommandParsing(string arguments) + { + var (directStatements, directParser) = ParseWithParser($"dir {arguments}"); + var (execStatements, execParser) = ParseWithParser($"exec $cmd {arguments}"); + + Assert.Empty(directParser.Errors); + Assert.Empty(execParser.Errors); + var direct = Assert.IsType(Assert.Single(directStatements)); + var exec = Assert.IsType(Assert.Single(execStatements)); + Assert.Equal(direct.Arguments.Count, exec.Arguments.Count); + for (int i = 0; i < direct.Arguments.Count; i++) + { + Assert.Equal(direct.Arguments[i].GetType(), exec.Arguments[i].GetType()); + Assert.Equal(CommandArgumentFormatter.Format(direct.Arguments[i]), CommandArgumentFormatter.Format(exec.Arguments[i])); + } + } + + [Theory] + [InlineData("exec \"echo\" --format=")] + [InlineData("exec \"echo\" --format=; echo after")] + public void Exec_MissingInlineOptionValue_ReportsError(string source) + { + var (_, parser) = ParseWithParser(source); + + Assert.NotEmpty(parser.Errors); + } + [Fact] public void Def_WithParenParameters_ParsesParameters() { diff --git a/CosmosDBShell.Tests/ScriptExecutionScopeTests.cs b/CosmosDBShell.Tests/ScriptExecutionScopeTests.cs index 2cb759b5..d7f048b0 100644 --- a/CosmosDBShell.Tests/ScriptExecutionScopeTests.cs +++ b/CosmosDBShell.Tests/ScriptExecutionScopeTests.cs @@ -213,16 +213,20 @@ public async Task RecursiveScript_StopsAtCallLimit_AndRestoresScope() } [Theory] - [InlineData(false)] - [InlineData(true)] - public async Task InvalidScript_DoesNotRegisterFunctions_AndRestoresScope(bool expression) + [InlineData(false, "if true {")] + [InlineData(true, "if true {")] + [InlineData(false, "for $x typo [1,2] { $value = 3 }")] + [InlineData(true, "for $x typo [1,2] { $value = 3 }")] + [InlineData(false, "do { $value = 3 } typo false")] + [InlineData(true, "do { $value = 3 } typo false")] + public async Task InvalidScript_DoesNotRegisterFunctions_AndRestoresScope(bool expression, string invalid) { var shell = ShellInterpreter.CreateInstance(); shell.SetVariable("value", new ShellNumber(1)); var path = Path.GetTempFileName(); try { - await File.WriteAllTextAsync(path, "def mustNotRegister { return 1 }; if true {", TestContext.Current.CancellationToken); + await File.WriteAllTextAsync(path, $"def mustNotRegister {{ return 1 }}; {invalid}", TestContext.Current.CancellationToken); var token = new Token(TokenType.Identifier, path, 0, path.Length); var state = expression ? await new CommandExpression(token).RunScriptAsync(shell, new(), CancellationToken.None) diff --git a/CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/ExpressionParser.cs b/CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/ExpressionParser.cs index 445100cb..55f6fab2 100644 --- a/CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/ExpressionParser.cs +++ b/CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/ExpressionParser.cs @@ -1722,7 +1722,7 @@ private Expression ParseFilterPathExpression(Token firstToken) } var indexToken = this.Consume(TokenType.Number, MessageService.GetString("expression_error_expected_array_index")); - int index = int.TryParse(indexToken.Value, NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsedIndex) ? parsedIndex : 0; + bool validIndex = int.TryParse(indexToken.Value, NumberStyles.Integer, CultureInfo.InvariantCulture, out var index); var indexedCloseBracket = this.Consume(TokenType.CloseBracket, MessageService.GetString("expression_error_expected_close_bracket")); var indexQuestionToken = this.TryConsumeQuestion(); end = indexedCloseBracket.Start + indexedCloseBracket.Length; @@ -1731,6 +1731,12 @@ private Expression ParseFilterPathExpression(Token firstToken) end = indexQuestionToken.Start + indexQuestionToken.Length; } + if (!validIndex) + { + this.ReportError(MessageService.GetArgsString("expression_error_invalid_number", "value", indexToken.Value), indexToken); + return new ErrorExpression(firstToken.Start, end - firstToken.Start); + } + segments.Add(new FilterIndexSegment(index, indexQuestionToken != null)); continue; } diff --git a/CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/ShellObject/ShellJson.cs b/CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/ShellObject/ShellJson.cs index ca9dd00c..0efb5b1a 100644 --- a/CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/ShellObject/ShellJson.cs +++ b/CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/ShellObject/ShellJson.cs @@ -91,13 +91,13 @@ public override object ConvertShellObject(DataType type) case DataType.Decimal: if (this.Value.ValueKind == JsonValueKind.Number) { - return this.Value.GetDecimal(); + return this.Value.GetDouble(); } if (this.Value.ValueKind == JsonValueKind.String) { var strValue = this.Value.GetString() ?? string.Empty; - if (decimal.TryParse(strValue, NumberStyles.Float, CultureInfo.InvariantCulture, out decimal decimalValue)) + if (double.TryParse(strValue, NumberStyles.Float, CultureInfo.InvariantCulture, out double decimalValue)) { return decimalValue; } diff --git a/CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/StatementParser.cs b/CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/StatementParser.cs index b2299056..3dcd602a 100644 --- a/CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/StatementParser.cs +++ b/CosmosDBShell/Azure.Data.Cosmos.Shell.Parser/StatementParser.cs @@ -229,6 +229,17 @@ private void SkipWs() } } + private Token ConsumeKeyword(string keyword, string message) + { + var token = this.expressionParser.Consume(TokenType.Identifier, message); + if (token.Length > 0 && !token.Value.Equals(keyword, StringComparison.OrdinalIgnoreCase)) + { + this.ReportError(message, token); + } + + return token; + } + private Statement? ParseSingleStatement() { if (this.expressionParser.IsAtEnd) @@ -387,8 +398,7 @@ private void SkipWs() return null; } - var inToken = this.expressionParser.Current; - this.expressionParser.Consume(TokenType.Identifier, MessageService.GetString("statement_error_expected_in")); + var inToken = this.ConsumeKeyword("in", MessageService.GetString("statement_error_expected_in")); var collection = this.expressionParser.ParseExpression(); var statement = this.ParseStatement(); @@ -435,8 +445,7 @@ private void SkipWs() return null; } - var whileToken = this.expressionParser.Current; - this.expressionParser.Consume(TokenType.Identifier, MessageService.GetString("statement_error_expected_while")); + var whileToken = this.ConsumeKeyword("while", MessageService.GetString("statement_error_expected_while")); var condition = this.expressionParser.ParseExpression(); return new DoWhileStatement(doToken, statement, whileToken, condition); @@ -507,7 +516,12 @@ private void SkipWs() this.expressionParser.Current.Type != TokenType.CloseBrace && this.expressionParser.Current.Type != TokenType.Pipe) { - var arg = this.expressionParser.ParsePrimaryExpression(); + var arg = this.ParseCommandArgument(); + if (arg == null) + { + break; + } + arguments.Add(arg); } @@ -907,6 +921,56 @@ private void SkipWs() } } + private Expression? ParseCommandArgument() + { + var commandWordParser = this.CreateCommandShellWordParser(); + var current = this.expressionParser.Current; + if (current?.Type != TokenType.Minus || !commandWordParser.IsCommandOptionStart()) + { + return commandWordParser.ParseShellWord(); + } + + var optionStartToken = current; + this.expressionParser.Advance(); + + bool doubleDash = false; + if (!this.expressionParser.IsAtEnd && + this.expressionParser.Current?.Type == TokenType.Minus) + { + doubleDash = true; + this.expressionParser.Advance(); + } + + if (this.expressionParser.IsAtEnd || + this.expressionParser.Current is not { Type: TokenType.Identifier } optionNameToken) + { + this.ReportError( + MessageService.GetArgsString("statement_error_expected_option_name", "prefix", doubleDash ? "--" : "-"), + this.expressionParser.Current); + return null; + } + + this.expressionParser.Advance(); + + Token? separatorToken = null; + Expression? optionValue = null; + if (!this.expressionParser.IsAtEnd && + this.expressionParser.Current is { Type: TokenType.Colon or TokenType.Assignment } separator) + { + separatorToken = separator; + this.expressionParser.Advance(); + optionValue = commandWordParser.ParseShellWord(); + if (optionValue == null) + { + this.ReportError( + MessageService.GetArgsString("statement_error_invalid_option_value", "option", optionNameToken.Value), + this.expressionParser.Current); + } + } + + return new CommandOption(optionStartToken, optionNameToken, separatorToken, optionValue); + } + private CommandStatement? ParseCommandStatement(Token? optToken = null) { try @@ -947,83 +1011,13 @@ private void SkipWs() this.expressionParser.Current.Type != TokenType.RedirectAppendError && !this.IsStderrRedirectStart()) { - var commandWordParser = this.CreateCommandShellWordParser(); - - // In command mode an argument is either a structured option (`-name`, `--name[=:]value`) - // or a "shell word": a maximal run of adjacent simple tokens concatenated as text. - // Strings, interpolated strings, $-variables and (expr)/[..]/{..} drop into expression - // mode unchanged so existing scripting features keep working. - if (this.expressionParser.Current.Type == TokenType.Minus && - commandWordParser.IsCommandOptionStart()) + var arg = this.ParseCommandArgument(); + if (arg == null) { - var optionStartToken = this.expressionParser.Current; - this.expressionParser.Advance(); - - bool doubleDash = false; - if (!this.expressionParser.IsAtEnd && - this.expressionParser.Current != null && - this.expressionParser.Current.Type == TokenType.Minus) - { - doubleDash = true; - this.expressionParser.Advance(); - } - - if (this.expressionParser.IsAtEnd || - this.expressionParser.Current == null || - this.expressionParser.Current.Type != TokenType.Identifier) - { - this.ReportError( - MessageService.GetArgsString("statement_error_expected_option_name", "prefix", doubleDash ? "--" : "-"), - this.expressionParser.Current); - break; - } - - var optionNameToken = this.expressionParser.Current; - var optionName = optionNameToken.Value; - this.expressionParser.Advance(); - - Token? separatorToken = null; - Expression? optionValue = null; - if (!this.expressionParser.IsAtEnd && - this.expressionParser.Current != null && - (this.expressionParser.Current.Type == TokenType.Colon || - this.expressionParser.Current.Type == TokenType.Assignment)) - { - separatorToken = this.expressionParser.Current; - this.expressionParser.Advance(); - - if (!this.expressionParser.IsAtEnd && - this.expressionParser.Current != null) - { - optionValue = commandWordParser.ParseShellWord(); - - if (optionValue == null) - { - this.ReportError( - MessageService.GetArgsString("statement_error_invalid_option_value", "option", optionName), - this.expressionParser.Current); - } - } - else - { - this.ReportError( - MessageService.GetArgsString("statement_error_invalid_option_value", "option", optionName), - this.expressionParser.Current); - } - } - - command.Arguments.Add(new CommandOption(optionStartToken, optionNameToken, separatorToken, optionValue)); + break; } - else - { - var arg = commandWordParser.ParseShellWord(); - if (arg == null) - { - break; - } - command.Arguments.Add(arg); - } + command.Arguments.Add(arg); } while (!this.expressionParser.IsAtEnd && diff --git a/README.md b/README.md index d25bce44..ef82ed91 100644 --- a/README.md +++ b/README.md @@ -240,6 +240,8 @@ Scripts are parsed and validated before their statements execute. Functions pres Parser nesting, expression tree depth, and active function/script calls have fixed [resource limits](docs/programming.md#resource-limits). Limit violations fail with diagnostics rather than continuing recursive parsing or execution. +Loop headers reject misspelled `in` and `while` keywords before execution, and `filter` rejects overflowing array indexes rather than selecting element zero. JSON numeric strings use the same decimal conversion as shell strings. Dynamic `exec` calls accept normal command options, negative arguments, and shell words. See [programming](docs/programming.md) and the [filter language](docs/filter-v1-spec.md). + Script diagnostics preserve source files, runtime failure categories, and function/script call sites. The language server shares the runtime's control-flow and duplicate-parameter checks, recognizes document-local functions, and checks nested commands and options. Incorrect function argument counts produce usage exit code `2`. See [validation and errors](docs/programming.md#validation-and-errors). Loops and functions preserve JSON `null` values. Numeric conditions use the same zero/nonzero rule for shell values and JSON properties, including fractional numbers. See [value conversion rules](docs/programming.md#numbers). diff --git a/docs/filter-v1-spec.md b/docs/filter-v1-spec.md index 98a53c81..f1fdb29b 100644 --- a/docs/filter-v1-spec.md +++ b/docs/filter-v1-spec.md @@ -200,6 +200,7 @@ Examples: - If the input is an array and the index exists, the element is returned. - If the index is out of range, the result is `null`. +- Index literals must fit in a non-negative signed 32-bit integer (`0` through `2147483647`). Larger literals are parse errors, including with optional access; they are never substituted with index zero. - If the input is not an array, evaluation fails unless optional access is used. Examples: diff --git a/docs/programming.md b/docs/programming.md index f7d15ed5..2a87fc13 100644 --- a/docs/programming.md +++ b/docs/programming.md @@ -36,6 +36,8 @@ JSON construction preserves the distinction between integers and shell decimals. JSON object and array literals accept larger integer-form values and preserve them as JSON integers. +When a numeric operation converts text to a decimal value, JSON string properties use the same culture-invariant `double` conversion as shell strings. For example, `$object = {"value":"2.5"}; $result = 3.0 - $object.value` produces `0.5`, just like `3.0 - "2.5"`. The `+` operator still concatenates strings rather than converting them to numbers. + Numeric Boolean conversion uses zero versus nonzero, including for fractional and large JSON numbers. JSON numbers use the same `double` conversion as decimal shell values for this check, so `if 1.5` and `if $object.value` behave alike when the property contains `1.5`. JSON `null` remains JSON `null` when bound by a `for` loop or passed through a function. Rebuilding an array from that value produces `[null]`, not `["null"]`. Text conversion remains explicit and separate from JSON type preservation. @@ -122,6 +124,8 @@ parameters = "[", { name }, "]" | "(", [ name, { ",", name } ], ")" ; Each command text or script file is fully parsed and checked for invalid control-flow placement and duplicate function parameters before any of its statements execute. Syntax or semantic errors prevent execution of that entire input. Script files are checked when invoked, including calls through `exec` and command expressions; callers are not recursively preflighted against dynamically selected files. +Loop headers require the case-insensitive keywords `in` in a `for` loop and `while` after a `do` body. A misspelled keyword is a syntax error; no statements in that input execute. + Runtime failures stop execution but do not roll back earlier successful commands. A failed command expression propagates an error rather than silently producing an empty result. Cancellation is checked between block statements and loop iterations, including loops without database commands. Host-requested cancellation propagates through script files, blocks, loops, and function calls without being converted into a positional runtime error. The shell reports a neutral result and records cancellation in the diagnostic log; call scopes and source context are restored. Cancellation exceptions without a canceled host token still follow the existing error/timeout handling. @@ -304,6 +308,7 @@ Notes: - If the evaluated value is a file path that exists, the shell runs it as a `.csh` script. - Argument parsing stops at `;`, newline, `}`, or `|` (so you can chain with pipes). +- Arguments use normal command syntax, including `-name` / `--name` options with space-separated, `=`, or `:` values, negative arguments such as `-5`, and unquoted shell words such as URLs and file patterns. Variables, JSON values, and parenthesized expressions retain their normal evaluation behavior. Quote a dash-prefixed word when it should be a positional string rather than an option. Examples: