precision Archives - IT Nursery

floating-point, Programming

IT Nursery

What’s the difference between a single precision and double precision floating point operation?

What is the difference between a single precision floating point operation and double precision floating operation? I’m especially interested in practical terms in ...

June 4, 2022
0 Comments

C#, Programming

IT Nursery

Why is a round-trip conversion via a string not safe for a double?

Recently I have had to serialize a double into text, and then get it back. The value seems to not be equivalent: double ...

June 3, 2022
0 Comments

PHP, Programming

IT Nursery

Unexpected results when working with very big integers on interpreted languages

I am trying to get the sum of 1 + 2 + ... + 1000000000, but I’m getting funny results in PHP and ...

June 2, 2022
0 Comments

dart, Programming

IT Nursery

How do you round a double in Dart to a given degree of precision AFTER the decimal point?

Given a double, I want to round it to a given number of points of precision after the decimal point, similar to PHP’s ...

May 28, 2022
0 Comments

Why are floating point numbers inaccurate?

Why do some numbers lose accuracy when stored as floating point numbers? For example, the decimal number 9.2 can be expressed exactly as ...

May 24, 2022
0 Comments

Get DateTime.Now with milliseconds precision

How can I exactly construct a time stamp of actual time with milliseconds precision? I need something like 16.4.2013 9:48:00:123. Is this possible? ...

May 20, 2022
0 Comments

How do I interpret precision and scale of a number in a database?

I have the following column specified in a database: decimal(5,2) How does one interpret this? According to the properties on the column as ...

May 19, 2022
0 Comments

How do I print a double value with full precision using cout?

In my earlier question I was printing a double using cout that got rounded when I wasn’t expecting it. How can I make ...

May 7, 2022
0 Comments

JavaScript displaying a float to 2 decimal places

I wanted to display a number to 2 decimal places. I thought I could use toPrecision(2) in JavaScript . However, if the number ...

May 7, 2022
0 Comments

What is the difference between float and double?

I’ve read about the difference between double precision and single precision. However, in most cases, float and double seem to be interchangeable, i.e. ...

May 3, 2022
0 Comments