What is the difference between a single precision floating point operation and double precision floating operation? I’m especially interested in practical terms in relation to video game consoles. For example, does the Nintendo 64 have a 64 bit processor and if it does then would that mean it was capable of double precision floating point … Read more
Recently I have had to serialize a double into text, and then get it back. The value seems to not be equivalent: double d1 = 0.84551240822557006; string s = d1.ToString(“R”); double d2 = double.Parse(s); bool s1 = d1 == d2; // -> s1 is False But according to MSDN: Standard Numeric Format Strings, the “R” … Read more
I am trying to get the sum of 1 + 2 + … + 1000000000, but I’m getting funny results in PHP and Node.js. PHP $sum = 0; for($i = 0; $i <= 1000000000 ; $i++) { $sum += $i; } printf(“%s”, number_format($sum, 0, “”, “”)); // 500000000067108992 Node.js var sum = 0; for (i … Read more
Given a double, I want to round it to a given number of points of precision after the decimal point, similar to PHP’s round() function. The closest thing I can find in the Dart docs is double.toStringAsPrecision(), but this is not quite what I need because it includes the digits before the decimal point in … Read more
Why do some numbers lose accuracy when stored as floating point numbers? For example, the decimal number 9.2 can be expressed exactly as a ratio of two decimal integers (92/10), both of which can be expressed exactly in binary (0b1011100/0b1010). However, the same ratio stored as a floating point number is never exactly equal to … Read more
How can I exactly construct a time stamp of actual time with milliseconds precision? I need something like 16.4.2013 9:48:00:123. Is this possible? I have an application, where I sample values 10 times per second, and I need to show them in a graph. 12 Answers 12
I have the following column specified in a database: decimal(5,2) How does one interpret this? According to the properties on the column as viewed in SQL Server Management studio I can see that it means: decimal(Numeric precision, Numeric scale). What do precision and scale mean in real terms? It would be easy to interpret this … Read more
In my earlier question I was printing a double using cout that got rounded when I wasn’t expecting it. How can I make cout print a double using full precision? 16 Answers 16
I wanted to display a number to 2 decimal places. I thought I could use toPrecision(2) in JavaScript . However, if the number is 0.05, I get 0.0500. I’d rather it stay the same. See it on JSbin. What is the best way to do this? I can think of coding a few solutions, but … Read more
I’ve read about the difference between double precision and single precision. However, in most cases, float and double seem to be interchangeable, i.e. using one or the other does not seem to affect the results. Is this really the case? When are floats and doubles interchangeable? What are the differences between them? 14 s 14 … Read more
